2024-04-16T07:28:53.093 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T07:28:53.093 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} --name matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi --verbose --owner scheduled_matan@teuthology --archive /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962 -- /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962/orig.config.yaml 2024-04-16T07:28:53.136 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T07:28:53.224 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962 branch: wip-matanb-crimson-replicated-april-15 description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} email: null first_in_suite: false job_id: '7657962' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-replicated-april-15 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: wip-matanb-crimson-replicated-april-15 env: CRIMSON_COMPAT: '1' sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 7594 sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-replicated-april-15 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA5UD4iWmcHqD0asquxePKd1zDgMEmVjKTp6tq2G1cvMf5S0DQQU9rBSKz2/86Sc2+mdSYsp1sjTz8YVmxtua0Y= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvF4CbbQpotyRwsdFpVtdhZM8X38sysgVEwOX2Waa1gG2kMjIKBCYqQIJe2ufBRAsWXlIguOU6QCW3lf1wy3uI= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_07:22:04 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T07:28:53.224 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa; will attempt to use it 2024-04-16T07:28:53.225 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks 2024-04-16T07:28:53.225 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T07:28:53.226 INFO:teuthology.task.internal:Checking packages... 2024-04-16T07:28:53.250 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '30b0aa0009a925d6a46536e55c5656edcd51ecdb' 2024-04-16T07:28:53.250 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:28:53.250 INFO:teuthology.packaging:ref: None 2024-04-16T07:28:53.250 INFO:teuthology.packaging:tag: None 2024-04-16T07:28:53.250 INFO:teuthology.packaging:branch: wip-matanb-crimson-replicated-april-15 2024-04-16T07:28:53.250 INFO:teuthology.packaging:sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:28:53.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-replicated-april-15 2024-04-16T07:28:53.476 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3109.g30b0aa00 2024-04-16T07:28:53.478 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T07:28:53.485 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T07:28:53.485 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T07:28:53.527 INFO:teuthology.task.internal:Saving configuration 2024-04-16T07:28:53.541 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T07:28:53.552 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:28:53.578 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:08.204180', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA5UD4iWmcHqD0asquxePKd1zDgMEmVjKTp6tq2G1cvMf5S0DQQU9rBSKz2/86Sc2+mdSYsp1sjTz8YVmxtua0Y='} 2024-04-16T07:28:53.600 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:08.202417', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvF4CbbQpotyRwsdFpVtdhZM8X38sysgVEwOX2Waa1gG2kMjIKBCYqQIJe2ufBRAsWXlIguOU6QCW3lf1wy3uI='} 2024-04-16T07:28:53.600 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T07:28:53.608 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-04-16T07:28:53.608 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-16T07:28:53.608 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T07:28:53.692 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3ace225e50>, signals=[15]) 2024-04-16T07:28:53.693 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T07:28:53.702 INFO:teuthology.task.internal:Opening connections... 2024-04-16T07:28:53.702 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-04-16T07:28:53.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:28:53.792 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2024-04-16T07:28:53.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:28:53.880 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T07:28:53.889 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-04-16T07:28:53.906 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-04-16T07:28:53.906 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-04-16T07:28:53.963 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-04-16T07:28:53.963 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-04-16T07:28:53.963 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-04-16T07:28:53.963 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T07:28:53.964 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T07:28:53.965 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-04-16T07:28:53.990 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2024-04-16T07:28:54.007 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2024-04-16T07:28:54.007 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2024-04-16T07:28:54.062 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2024-04-16T07:28:54.062 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="9" 2024-04-16T07:28:54.062 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="9" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el9" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:LOGO="fedora-logo-icon" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T07:28:54.063 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T07:28:54.063 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-04-16T07:28:54.081 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T07:28:54.095 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T07:28:54.113 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T07:28:54.113 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-04-16T07:28:54.116 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2024-04-16T07:28:54.131 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T07:28:54.143 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T07:28:54.144 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-04-16T07:28:54.173 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2024-04-16T07:28:54.264 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T07:28:54.321 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T07:28:54.340 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-16T07:28:54.340 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-16T07:28:54.340 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T07:28:54.341 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T07:28:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:28:54.356 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-04-16T07:28:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:28:54.357 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-04-16T07:28:54.412 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-361.el9.x86_64 2024-04-16T07:28:54.413 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.14.0-361.el9.x86_64 2024-04-16T07:28:54.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2024-04-16T07:28:54.414 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-361.el9.x86_64 2024-04-16T07:28:54.414 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-361.el9.x86_64 2024-04-16T07:28:54.414 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-04-16T07:28:56.530 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-16T07:28:56.561 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-16T07:28:57.947 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-04-16T07:28:57.953 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-04-16T07:28:58.793 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-16T07:28:58.812 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-16T07:29:00.795 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-16T07:29:07.386 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.2 MB 00:03 2024-04-16T07:29:08.779 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 16 kB 00:00 2024-04-16T07:29:09.805 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-04-16T07:29:14.905 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-04-16T07:29:17.219 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T07:29:17.219 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T07:29:17.309 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:29:17.318 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:29:17.318 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-04-16T07:29:17.319 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:29:17.320 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 77 M 2024-04-16T07:29:17.320 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 123 M 2024-04-16T07:29:17.320 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:29:17.481 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.7 MB/s | 277 kB 00:00 2024-04-16T07:29:18.131 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-04-16T07:29:19.106 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 19 MB/s | 34 MB 00:01 2024-04-16T07:29:19.765 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 12 MB/s | 28 MB 00:02 2024-04-16T07:29:19.767 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:29:19.768 INFO:teuthology.orchestra.run.smithi079.stdout:Total 31 MB/s | 77 MB 00:02 2024-04-16T07:29:19.768 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:29:20.146 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:29:20.146 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:29:21.001 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:29:21.001 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:29:23.282 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:29:23.644 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 815 kB/s | 19 MB 00:23 2024-04-16T07:29:23.877 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:29:23.964 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:26.685 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:26.823 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:28.087 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-16T07:29:29.589 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-04-16T07:29:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:31.509 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-16T07:29:31.531 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:29:36.261 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:29:36.622 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-04-16T07:29:38.996 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T07:29:38.996 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T07:29:39.097 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:29:39.106 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:29:39.107 INFO:teuthology.orchestra.run.smithi167.stdout:Install 4 Packages 2024-04-16T07:29:39.108 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:29:39.108 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 77 M 2024-04-16T07:29:39.108 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 123 M 2024-04-16T07:29:39.108 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:29:39.290 INFO:teuthology.orchestra.run.smithi167.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.5 MB/s | 277 kB 00:00 2024-04-16T07:29:40.732 INFO:teuthology.orchestra.run.smithi167.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 9.3 MB/s | 15 MB 00:01 2024-04-16T07:29:41.107 INFO:teuthology.orchestra.run.smithi167.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-04-16T07:29:41.499 INFO:teuthology.orchestra.run.smithi167.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 13 MB/s | 28 MB 00:02 2024-04-16T07:29:41.503 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:29:41.503 INFO:teuthology.orchestra.run.smithi167.stdout:Total 32 MB/s | 77 MB 00:02 2024-04-16T07:29:41.503 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:29:41.879 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:29:41.880 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:29:42.702 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:29:42.702 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:29:45.020 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:29:45.638 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:29:45.726 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:48.482 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:29:48.620 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:29:53.461 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:29:58.220 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:05.852 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:05.852 INFO:teuthology.orchestra.run.smithi079.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T07:30:05.853 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:30:05.895 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:06.300 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:06.300 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:30:06.300 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:30:06.301 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:30:06.721 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:06.721 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:30:06.721 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:30:06.721 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:06.722 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:06.722 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T07:30:06.722 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:06.722 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:30:06.722 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:30:07.065 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:07.161 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:07.161 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:07.162 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T07:30:07.162 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T07:30:07.162 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T07:30:07.162 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-16T07:30:07.162 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T07:30:07.162 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:07.242 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:07.243 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:07.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2-tools 2024-04-16T07:30:07.281 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2-tools 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.06 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 68.el9 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Group : Unspecified 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 8264254 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T07:30:07.282 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : builder@centos.org 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Support tools for GRUB. 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:30:07.283 INFO:teuthology.orchestra.run.smithi079.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T07:30:07.284 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T07:30:07.284 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 5.14.0-438.el9.x86_64 2024-04-16T07:30:07.284 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T07:30:07.873 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-04-16T07:30:09.298 INFO:teuthology.orchestra.run.smithi079.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T07:30:09.333 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-04-16T07:30:09.336 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-04-16T07:30:09.350 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.oNqZ7duC3Q 2024-04-16T07:30:09.350 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oNqZ7duC3Q 2024-04-16T07:30:09.421 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.oNqZ7duC3Q 2024-04-16T07:30:09.559 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.oNqZ7duC3Q is 6KB 2024-04-16T07:30:09.577 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.oNqZ7duC3Q 2024-04-16T07:30:09.591 DEBUG:teuthology.orchestra.run.smithi079:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T07:30:09.656 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T07:30:09.656 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T07:30:09.656 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T07:30:09.656 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T07:30:09.658 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T07:30:09.809 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-04-16T07:30:27.469 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:27.469 INFO:teuthology.orchestra.run.smithi167.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T07:30:27.469 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:30:27.513 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:27.932 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:27.932 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:30:27.932 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:30:27.932 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:30:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:30:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:30:28.399 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:30:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T07:30:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T07:30:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:30:28.399 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:30:28.724 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:28.862 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:28.862 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:28.863 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T07:30:28.863 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T07:30:28.863 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T07:30:28.863 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-04-16T07:30:28.863 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T07:30:28.863 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:30:28.943 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:30:28.943 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:30:28.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo rpm -qi grub2-tools 2024-04-16T07:30:28.977 INFO:teuthology.orchestra.run.smithi167.stdout:Name : grub2-tools 2024-04-16T07:30:28.977 INFO:teuthology.orchestra.run.smithi167.stdout:Epoch : 1 2024-04-16T07:30:28.977 INFO:teuthology.orchestra.run.smithi167.stdout:Version : 2.06 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Release : 68.el9 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: x86_64 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Group : Unspecified 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Size : 8264254 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:License : GPLv3+ 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Packager : builder@centos.org 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:Vendor : CentOS 2024-04-16T07:30:28.978 INFO:teuthology.orchestra.run.smithi167.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T07:30:28.979 INFO:teuthology.orchestra.run.smithi167.stdout:Summary : Support tools for GRUB. 2024-04-16T07:30:28.979 INFO:teuthology.orchestra.run.smithi167.stdout:Description : 2024-04-16T07:30:28.979 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:30:28.979 INFO:teuthology.orchestra.run.smithi167.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T07:30:28.979 INFO:teuthology.orchestra.run.smithi167.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T07:30:28.979 INFO:teuthology.orchestra.run.smithi167.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T07:30:28.979 INFO:teuthology.orchestra.run.smithi167.stdout:hardware devices. 2024-04-16T07:30:28.979 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:30:28.979 INFO:teuthology.orchestra.run.smithi167.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T07:30:28.980 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T07:30:28.980 INFO:teuthology.task.kernel:Updating grub on smithi167 to boot 5.14.0-438.el9.x86_64 2024-04-16T07:30:28.980 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T07:30:29.665 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2024-04-16T07:30:31.076 INFO:teuthology.orchestra.run.smithi167.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T07:30:31.120 INFO:teuthology.orchestra.run.smithi167.stderr:done 2024-04-16T07:30:31.123 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-04-16T07:30:31.179 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.AaTG7MVuMd 2024-04-16T07:30:31.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AaTG7MVuMd 2024-04-16T07:30:31.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.AaTG7MVuMd 2024-04-16T07:30:31.395 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.AaTG7MVuMd is 6KB 2024-04-16T07:30:31.414 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.AaTG7MVuMd 2024-04-16T07:30:31.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T07:30:31.492 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T07:30:31.492 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T07:30:31.492 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T07:30:31.493 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T07:30:31.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T07:30:31.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2024-04-16T07:30:39.812 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-16T07:30:39.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-16T07:30:39.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:01.686 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-04-16T07:31:01.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-16T07:31:01.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:20.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-16T07:31:29.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-16T07:31:29.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:32.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-16T07:31:39.823 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T07:31:44.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-16T07:31:44.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:48.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-16T07:31:48.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:31:49.412 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-04-16T07:31:49.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-16T07:31:49.885 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T07:31:49.886 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-04-16T07:31:49.901 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-438.el9.x86_64 2024-04-16T07:31:49.901 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T07:31:49.901 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:31:49.901 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:31:50.902 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-16T07:31:50.903 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:31:51.013 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-04-16T07:31:51.040 DEBUG:teuthology.parallel:result is None 2024-04-16T07:32:02.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-16T07:32:17.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-16T07:32:17.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:32:20.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-16T07:32:38.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-16T07:32:38.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:32:39.193 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-04-16T07:32:39.561 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-16T07:32:39.562 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-438.el9.x86_64"... 2024-04-16T07:32:39.562 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-04-16T07:32:39.579 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-438.el9.x86_64 2024-04-16T07:32:39.580 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T07:32:39.580 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:32:39.580 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:32:40.581 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-16T07:32:40.581 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:32:40.650 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2024-04-16T07:32:40.685 DEBUG:teuthology.parallel:result is None 2024-04-16T07:32:40.686 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T07:32:40.695 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T07:32:40.695 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:32:40.699 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:32:40.717 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T07:32:40.738 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T07:32:40.771 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T07:32:40.771 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:32:40.774 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:32:40.832 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T07:32:40.841 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T07:32:40.841 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:32:40.844 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:32:40.882 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:32:40.884 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:32:40.896 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:32:40.898 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:32:40.900 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T07:32:40.909 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T07:32:40.909 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:32:40.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:32:40.974 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T07:32:40.985 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T07:32:40.985 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:32:41.015 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:32:41.032 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:32:41.116 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:32:41.169 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:32:41.253 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:32:41.299 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:32:41.299 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:32:41.368 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:32:41.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:32:41.468 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:32:41.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:32:41.601 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:32:41.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:32:41.672 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-04-16T07:32:41.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2024-04-16T07:32:41.731 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:32:41.762 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:32:42.125 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T07:32:42.134 INFO:teuthology.task.internal:Starting timer... 2024-04-16T07:32:42.134 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T07:32:42.166 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T07:32:42.198 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T07:32:42.199 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-04-16T07:32:42.258 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-04-16T07:32:42.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo service auditd rotate 2024-04-16T07:32:42.323 INFO:teuthology.orchestra.run.smithi167.stdout:Rotating logs: 2024-04-16T07:32:42.325 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T07:32:42.325 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-04-16T07:32:42.352 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-04-16T07:32:42.352 DEBUG:teuthology.orchestra.run.smithi167:> /usr/sbin/getenforce 2024-04-16T07:32:42.376 INFO:teuthology.orchestra.run.smithi167.stdout:Permissive 2024-04-16T07:32:42.376 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi167.front.sepia.ceph.com': 'permissive'} 2024-04-16T07:32:42.376 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:32:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:32:42.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:32:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:32:42.452 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T07:32:42.452 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:32:42.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:32:42.527 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T07:32:42.544 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T07:32:42.563 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T07:32:42.564 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T07:37:12.323 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2024-04-16T07:37:12.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-16T07:37:12.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:37:12.440 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-04-16T07:37:12.529 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-16T07:37:12.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-16T07:37:12.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:37:12.628 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-04-16T07:37:12.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-16T07:37:12.711 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T07:37:12.727 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T07:37:12.728 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:37:12.729 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:37:12.732 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:37:12.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:37:12.765 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:37:12.784 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:37:12.790 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:37:12.808 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:37:12.815 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-04-16T07:37:12.830 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-04-16T07:37:12.839 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: ntpd: command not found 2024-04-16T07:37:12.849 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout:506 Cannot talk to daemon 2024-04-16T07:37:12.866 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:37:12.871 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:37:12.889 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:37:12.919 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:37:12.924 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:37:12.924 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-04-16T07:37:12.924 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:12.924 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:12.924 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:12.924 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:12.945 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:37:12.949 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:37:12.949 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-04-16T07:37:12.949 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:12.949 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:12.949 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:12.949 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:37:12.949 INFO:teuthology.run_tasks:Running task install... 2024-04-16T07:37:12.961 DEBUG:teuthology.task.install:project ceph 2024-04-16T07:37:12.961 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}} 2024-04-16T07:37:12.961 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'} 2024-04-16T07:37:12.962 INFO:teuthology.task.install:Using flavor: crimson 2024-04-16T07:37:12.969 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T07:37:12.969 INFO:teuthology.task.install:extra packages: [] 2024-04-16T07:37:12.969 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'tag': None, 'wait_for_package': False} 2024-04-16T07:37:12.970 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:37:12.973 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'tag': None, 'wait_for_package': False} 2024-04-16T07:37:12.973 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:37:13.131 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/ 2024-04-16T07:37:13.131 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3109.g30b0aa00 2024-04-16T07:37:13.140 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/ 2024-04-16T07:37:13.140 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3109.g30b0aa00 2024-04-16T07:37:13.451 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:37:13.451 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:37:13.451 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:37:13.489 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:37:13.490 DEBUG:teuthology.orchestra.run.smithi167:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/30b0aa0009a925d6a46536e55c5656edcd51ecdb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:37:13.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:37:13.646 DEBUG:teuthology.orchestra.run.smithi167:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:37:13.715 INFO:teuthology.orchestra.run.smithi167.stdout:check_obsoletes = 1 2024-04-16T07:37:13.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum clean all 2024-04-16T07:37:13.968 INFO:teuthology.orchestra.run.smithi167.stdout:74 files removed 2024-04-16T07:37:13.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-radosgw 2024-04-16T07:37:15.192 INFO:teuthology.orchestra.run.smithi167.stdout:ceph packages for x86_64 97 kB/s | 87 kB 00:00 2024-04-16T07:37:16.538 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:37:16.538 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:37:16.538 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:37:16.574 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:37:16.575 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/30b0aa0009a925d6a46536e55c5656edcd51ecdb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:37:16.648 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:37:16.733 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:37:16.762 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-04-16T07:37:16.763 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-04-16T07:37:16.867 INFO:teuthology.orchestra.run.smithi167.stdout:ceph noarch packages 7.2 kB/s | 12 kB 00:01 2024-04-16T07:37:17.007 INFO:teuthology.orchestra.run.smithi079.stdout:74 files removed 2024-04-16T07:37:17.035 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2024-04-16T07:37:17.609 INFO:teuthology.orchestra.run.smithi167.stdout:ceph source packages 2.9 kB/s | 2.0 kB 00:00 2024-04-16T07:37:18.078 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-04-16T07:37:18.142 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 108 kB/s | 87 kB 00:00 2024-04-16T07:37:19.126 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 12 kB/s | 12 kB 00:00 2024-04-16T07:37:19.861 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-04-16T07:37:20.552 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 1.5 kB/s | 2.0 kB 00:01 2024-04-16T07:37:21.396 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 8.3 MB/s | 6.7 MB 00:00 2024-04-16T07:37:22.815 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-04-16T07:37:23.695 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-16T07:37:23.744 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-04-16T07:37:25.445 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-04-16T07:37:29.488 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-16T07:37:30.261 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 3.5 MB/s | 19 MB 00:05 2024-04-16T07:37:31.139 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-16T07:37:33.040 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-16T07:37:34.388 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.2 MB 00:00 2024-04-16T07:37:35.997 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-16T07:37:37.048 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-16T07:37:38.098 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-16T07:37:40.960 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:37:40.971 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:37:40.971 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:37:40.971 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 10 M 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrading: 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: librados2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.7 M 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.9 M 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 19 M 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 25 k 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:37:40.972 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 797 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 546 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.2 M 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse 2024-04-16T07:37:40.973 INFO:teuthology.orchestra.run.smithi167.stdout: x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 45 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 135 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 159 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 315 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 293 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 99 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout:Installing weak dependencies: 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:37:40.974 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:37:40.975 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:37:40.975 INFO:teuthology.orchestra.run.smithi167.stdout:Install 30 Packages 2024-04-16T07:37:40.975 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrade 2 Packages 2024-04-16T07:37:40.975 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:37:40.975 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 59 M 2024-04-16T07:37:40.975 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:37:42.022 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-16T07:37:44.843 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:37:44.846 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:37:44.846 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:37:44.846 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 10 M 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.7 M 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.9 M 2024-04-16T07:37:44.847 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 19 M 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 25 k 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 797 k 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 546 k 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:37:44.848 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.2 M 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 45 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 135 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 159 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 315 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 293 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 99 k 2024-04-16T07:37:44.849 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout:Installing weak dependencies: 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout:Install 30 Packages 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-04-16T07:37:44.850 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:37:44.852 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 59 M 2024-04-16T07:37:44.852 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:37:44.907 INFO:teuthology.orchestra.run.smithi167.stdout:(1/32): ceph-radosgw-19.0.0-3109.g30b0aa00.el9. 2.7 MB/s | 10 MB 00:03 2024-04-16T07:37:44.991 INFO:teuthology.orchestra.run.smithi167.stdout:(2/32): ceph-selinux-19.0.0-3109.g30b0aa00.el9. 303 kB/s | 25 kB 00:00 2024-04-16T07:37:45.700 INFO:teuthology.orchestra.run.smithi167.stdout:(3/32): libcephfs2-19.0.0-3109.g30b0aa00.el9.x8 1.1 MB/s | 797 kB 00:00 2024-04-16T07:37:46.250 INFO:teuthology.orchestra.run.smithi167.stdout:(4/32): libradosstriper1-19.0.0-3109.g30b0aa00. 993 kB/s | 546 kB 00:00 2024-04-16T07:37:48.292 INFO:teuthology.orchestra.run.smithi167.stdout:(5/32): ceph-base-19.0.0-3109.g30b0aa00.el9.x86 853 kB/s | 5.9 MB 00:07 2024-04-16T07:37:48.401 INFO:teuthology.orchestra.run.smithi167.stdout:(6/32): python3-ceph-argparse-19.0.0-3109.g30b0 417 kB/s | 45 kB 00:00 2024-04-16T07:37:48.701 INFO:teuthology.orchestra.run.smithi167.stdout:(7/32): python3-ceph-common-19.0.0-3109.g30b0aa 449 kB/s | 135 kB 00:00 2024-04-16T07:37:48.831 INFO:teuthology.orchestra.run.smithi079.stdout:(1/32): ceph-radosgw-19.0.0-3109.g30b0aa00.el9. 2.8 MB/s | 10 MB 00:03 2024-04-16T07:37:48.931 INFO:teuthology.orchestra.run.smithi079.stdout:(2/32): ceph-selinux-19.0.0-3109.g30b0aa00.el9. 252 kB/s | 25 kB 00:00 2024-04-16T07:37:48.951 INFO:teuthology.orchestra.run.smithi167.stdout:(8/32): python3-cephfs-19.0.0-3109.g30b0aa00.el 638 kB/s | 159 kB 00:00 2024-04-16T07:37:49.208 INFO:teuthology.orchestra.run.smithi079.stdout:(3/32): libcephfs2-19.0.0-3109.g30b0aa00.el9.x8 2.8 MB/s | 797 kB 00:00 2024-04-16T07:37:49.382 INFO:teuthology.orchestra.run.smithi079.stdout:(4/32): libradosstriper1-19.0.0-3109.g30b0aa00. 3.1 MB/s | 546 kB 00:00 2024-04-16T07:37:49.590 INFO:teuthology.orchestra.run.smithi079.stdout:(5/32): ceph-base-19.0.0-3109.g30b0aa00.el9.x86 1.4 MB/s | 5.9 MB 00:04 2024-04-16T07:37:49.685 INFO:teuthology.orchestra.run.smithi167.stdout:(9/32): python3-rados-19.0.0-3109.g30b0aa00.el9 430 kB/s | 315 kB 00:00 2024-04-16T07:37:49.690 INFO:teuthology.orchestra.run.smithi079.stdout:(6/32): python3-ceph-argparse-19.0.0-3109.g30b0 452 kB/s | 45 kB 00:00 2024-04-16T07:37:49.808 INFO:teuthology.orchestra.run.smithi079.stdout:(7/32): python3-ceph-common-19.0.0-3109.g30b0aa 1.1 MB/s | 135 kB 00:00 2024-04-16T07:37:50.108 INFO:teuthology.orchestra.run.smithi079.stdout:(8/32): python3-cephfs-19.0.0-3109.g30b0aa00.el 532 kB/s | 159 kB 00:00 2024-04-16T07:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout:(10/32): python3-rbd-19.0.0-3109.g30b0aa00.el9. 445 kB/s | 293 kB 00:00 2024-04-16T07:37:50.433 INFO:teuthology.orchestra.run.smithi079.stdout:(9/32): librgw2-19.0.0-3109.g30b0aa00.el9.x86_6 5.0 MB/s | 5.2 MB 00:01 2024-04-16T07:37:50.466 INFO:teuthology.orchestra.run.smithi079.stdout:(10/32): python3-rados-19.0.0-3109.g30b0aa00.el 879 kB/s | 315 kB 00:00 2024-04-16T07:37:50.542 INFO:teuthology.orchestra.run.smithi079.stdout:(11/32): python3-rbd-19.0.0-3109.g30b0aa00.el9. 2.6 MB/s | 293 kB 00:00 2024-04-16T07:37:50.576 INFO:teuthology.orchestra.run.smithi079.stdout:(12/32): python3-rgw-19.0.0-3109.g30b0aa00.el9. 914 kB/s | 99 kB 00:00 2024-04-16T07:37:50.593 INFO:teuthology.orchestra.run.smithi167.stdout:(11/32): python3-rgw-19.0.0-3109.g30b0aa00.el9. 397 kB/s | 99 kB 00:00 2024-04-16T07:37:50.701 INFO:teuthology.orchestra.run.smithi079.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 668 kB/s | 106 kB 00:00 2024-04-16T07:37:50.710 INFO:teuthology.orchestra.run.smithi167.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 909 kB/s | 106 kB 00:00 2024-04-16T07:37:50.735 INFO:teuthology.orchestra.run.smithi079.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-04-16T07:37:50.760 INFO:teuthology.orchestra.run.smithi079.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-04-16T07:37:50.769 INFO:teuthology.orchestra.run.smithi167.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-04-16T07:37:50.803 INFO:teuthology.orchestra.run.smithi167.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-16T07:37:50.810 INFO:teuthology.orchestra.run.smithi079.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-04-16T07:37:50.835 INFO:teuthology.orchestra.run.smithi079.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-04-16T07:37:50.861 INFO:teuthology.orchestra.run.smithi079.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 832 kB/s | 41 kB 00:00 2024-04-16T07:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-04-16T07:37:50.886 INFO:teuthology.orchestra.run.smithi079.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-16T07:37:50.903 INFO:teuthology.orchestra.run.smithi167.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-04-16T07:37:50.911 INFO:teuthology.orchestra.run.smithi079.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-04-16T07:37:50.936 INFO:teuthology.orchestra.run.smithi079.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 922 kB/s | 46 kB 00:00 2024-04-16T07:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2024-04-16T07:37:51.028 INFO:teuthology.orchestra.run.smithi167.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2024-04-16T07:37:51.164 INFO:teuthology.orchestra.run.smithi167.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 260 kB/s | 35 kB 00:00 2024-04-16T07:37:51.205 INFO:teuthology.orchestra.run.smithi167.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-04-16T07:37:51.353 INFO:teuthology.orchestra.run.smithi079.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 699 kB/s | 309 kB 00:00 2024-04-16T07:37:51.429 INFO:teuthology.orchestra.run.smithi079.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-04-16T07:37:51.532 INFO:teuthology.orchestra.run.smithi079.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-04-16T07:37:51.630 INFO:teuthology.orchestra.run.smithi167.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 727 kB/s | 309 kB 00:00 2024-04-16T07:37:51.646 INFO:teuthology.orchestra.run.smithi079.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-04-16T07:37:51.671 INFO:teuthology.orchestra.run.smithi079.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 475 kB/s | 67 kB 00:00 2024-04-16T07:37:51.747 INFO:teuthology.orchestra.run.smithi079.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-16T07:37:51.855 INFO:teuthology.orchestra.run.smithi079.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-16T07:37:51.888 INFO:teuthology.orchestra.run.smithi079.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-04-16T07:37:51.947 INFO:teuthology.orchestra.run.smithi167.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-16T07:37:52.014 INFO:teuthology.orchestra.run.smithi167.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-04-16T07:37:52.081 INFO:teuthology.orchestra.run.smithi167.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 731 kB/s | 49 kB 00:00 2024-04-16T07:37:52.156 INFO:teuthology.orchestra.run.smithi167.stdout:(25/32): librgw2-19.0.0-3109.g30b0aa00.el9.x86_ 909 kB/s | 5.2 MB 00:05 2024-04-16T07:37:52.181 INFO:teuthology.orchestra.run.smithi167.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2024-04-16T07:37:52.265 INFO:teuthology.orchestra.run.smithi167.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-16T07:37:52.365 INFO:teuthology.orchestra.run.smithi167.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-16T07:37:52.665 INFO:teuthology.orchestra.run.smithi167.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2024-04-16T07:37:53.280 INFO:teuthology.orchestra.run.smithi079.stdout:(30/32): librbd1-19.0.0-3109.g30b0aa00.el9.x86_ 2.6 MB/s | 3.7 MB 00:01 2024-04-16T07:37:55.847 INFO:teuthology.orchestra.run.smithi079.stdout:(31/32): librados2-19.0.0-3109.g30b0aa00.el9.x8 871 kB/s | 3.4 MB 00:03 2024-04-16T07:37:56.791 INFO:teuthology.orchestra.run.smithi167.stdout:(30/32): ceph-common-19.0.0-3109.g30b0aa00.el9. 1.2 MB/s | 19 MB 00:15 2024-04-16T07:37:57.899 INFO:teuthology.orchestra.run.smithi167.stdout:(31/32): librbd1-19.0.0-3109.g30b0aa00.el9.x86_ 719 kB/s | 3.7 MB 00:05 2024-04-16T07:37:59.849 INFO:teuthology.orchestra.run.smithi167.stdout:(32/32): librados2-19.0.0-3109.g30b0aa00.el9.x8 465 kB/s | 3.4 MB 00:07 2024-04-16T07:37:59.854 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:37:59.854 INFO:teuthology.orchestra.run.smithi167.stdout:Total 3.1 MB/s | 59 MB 00:18 2024-04-16T07:37:59.855 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:37:59.894 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:37:59.894 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:38:01.900 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:38:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T07:38:02.526 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:02.633 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:02.724 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:38:02.761 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:02.824 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:02.865 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T07:38:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T07:38:02.925 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T07:38:02.947 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:38:02.967 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:38:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:38:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:38:03.379 INFO:teuthology.orchestra.run.smithi079.stdout:(32/32): ceph-common-19.0.0-3109.g30b0aa00.el9. 1.1 MB/s | 19 MB 00:18 2024-04-16T07:38:03.386 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:38:03.386 INFO:teuthology.orchestra.run.smithi079.stdout:Total 3.2 MB/s | 59 MB 00:18 2024-04-16T07:38:03.386 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:38:03.426 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:38:03.426 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:38:03.530 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T07:38:03.560 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:03.627 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 12/34 2024-04-16T07:38:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 14/34 2024-04-16T07:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T07:38:03.871 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:38:03.913 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:38:03.913 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:38:03.959 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:38:03.995 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T07:38:04.103 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T07:38:04.146 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T07:38:04.189 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T07:38:04.219 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T07:38:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T07:38:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T07:38:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T07:38:04.915 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T07:38:04.945 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:38:05.084 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 27/34 2024-04-16T07:38:05.114 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 28/34 2024-04-16T07:38:05.398 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:38:05.865 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T07:38:05.923 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:06.013 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:06.110 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:38:06.250 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:06.312 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:06.350 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T07:38:06.378 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T07:38:06.404 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T07:38:06.427 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:38:06.444 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:38:06.603 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:38:06.634 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:38:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T07:38:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:07.072 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:07.119 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 12/34 2024-04-16T07:38:07.177 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:38:07.203 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 14/34 2024-04-16T07:38:07.280 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T07:38:07.316 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:38:07.403 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:38:07.431 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T07:38:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:07.538 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T07:38:07.575 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T07:38:07.614 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T07:38:07.639 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T07:38:07.668 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T07:38:07.699 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T07:38:07.702 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:07.739 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T07:38:08.294 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T07:38:08.322 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:38:08.385 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:08.393 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:38:08.430 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:38:08.461 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 27/34 2024-04-16T07:38:08.491 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 28/34 2024-04-16T07:38:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:38:08.526 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:38:08.526 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:38:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:38:08.565 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:11.802 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:38:11.841 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:38:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:38:11.929 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:38:11.929 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:38:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:38:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /sys 2024-04-16T07:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /proc 2024-04-16T07:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /mnt 2024-04-16T07:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /var/tmp 2024-04-16T07:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /home 2024-04-16T07:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /root 2024-04-16T07:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /tmp 2024-04-16T07:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:38:19.323 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:38:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:38:19.348 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:38:19.348 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:38:19.348 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:38:19.348 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:38:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:38:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:38:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:38:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:20.681 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 6/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 7/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 8/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 9/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/34 2024-04-16T07:38:20.682 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 12/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T07:38:20.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T07:38:20.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T07:38:20.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T07:38:20.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T07:38:20.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T07:38:20.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T07:38:20.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:20.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T07:38:20.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/34 2024-04-16T07:38:21.736 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout:Upgraded: 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:38:21.737 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.738 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:38:21.739 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:38:21.863 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:21.863 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-04-16T07:38:21.863 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-04-16T07:38:21.864 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-04-16T07:38:21.864 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-04-16T07:38:21.864 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-04-16T07:38:21.864 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-04-16T07:38:21.864 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-04-16T07:38:21.864 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:38:21.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-test 2024-04-16T07:38:22.570 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:44 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:38:22.720 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:38:22.721 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:38:22.721 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:38:22.721 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:38:22.721 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 84 M 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 173 k 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T07:38:22.722 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:38:22.723 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:38:22.723 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:38:22.723 INFO:teuthology.orchestra.run.smithi167.stdout:Install 11 Packages 2024-04-16T07:38:22.723 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:38:22.723 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 87 M 2024-04-16T07:38:22.723 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 814 M 2024-04-16T07:38:22.723 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:38:22.899 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:38:22.925 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:38:22.925 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:38:22.925 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:38:22.925 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:38:22.926 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:38:22.926 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:38:22.927 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:38:23.010 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:38:23.010 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:23.686 INFO:teuthology.orchestra.run.smithi167.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-04-16T07:38:23.795 INFO:teuthology.orchestra.run.smithi167.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-04-16T07:38:23.837 INFO:teuthology.orchestra.run.smithi167.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-04-16T07:38:23.870 INFO:teuthology.orchestra.run.smithi167.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-04-16T07:38:23.904 INFO:teuthology.orchestra.run.smithi167.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.6 MB/s | 90 kB 00:00 2024-04-16T07:38:24.180 INFO:teuthology.orchestra.run.smithi167.stdout:(6/11): libcephsqlite-19.0.0-3109.g30b0aa00.el9 269 kB/s | 173 kB 00:00 2024-04-16T07:38:24.238 INFO:teuthology.orchestra.run.smithi167.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 561 kB/s | 187 kB 00:00 2024-04-16T07:38:24.243 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 6/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 7/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 8/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 9/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:38:24.244 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 12/34 2024-04-16T07:38:24.245 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:38:24.245 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T07:38:24.245 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T07:38:24.245 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T07:38:24.245 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T07:38:24.245 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T07:38:24.246 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T07:38:24.246 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T07:38:24.246 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T07:38:24.246 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T07:38:24.246 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T07:38:24.246 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T07:38:24.246 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T07:38:24.247 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T07:38:24.247 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T07:38:24.247 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T07:38:24.248 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T07:38:24.248 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T07:38:24.248 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:38:24.248 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T07:38:24.248 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/34 2024-04-16T07:38:24.372 INFO:teuthology.orchestra.run.smithi167.stdout:(8/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.4 MB/s | 469 kB 00:00 2024-04-16T07:38:24.397 INFO:teuthology.orchestra.run.smithi167.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-16T07:38:24.864 INFO:teuthology.orchestra.run.smithi167.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 248 kB/s | 122 kB 00:00 2024-04-16T07:38:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T07:38:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:38:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:38:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T07:38:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T07:38:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T07:38:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T07:38:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:38:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:38:25.463 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:38:25.700 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2024-04-16T07:38:26.272 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:44 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:38:26.427 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:38:26.428 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 84 M 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 173 k 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T07:38:26.429 INFO:teuthology.orchestra.run.smithi079.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:Install 11 Packages 2024-04-16T07:38:26.430 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:38:26.431 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 87 M 2024-04-16T07:38:26.431 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 814 M 2024-04-16T07:38:26.431 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:38:27.082 INFO:teuthology.orchestra.run.smithi079.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-04-16T07:38:27.291 INFO:teuthology.orchestra.run.smithi079.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 4.9 MB/s | 1.0 MB 00:00 2024-04-16T07:38:27.375 INFO:teuthology.orchestra.run.smithi079.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-04-16T07:38:27.425 INFO:teuthology.orchestra.run.smithi079.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-16T07:38:27.475 INFO:teuthology.orchestra.run.smithi079.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.8 MB/s | 90 kB 00:00 2024-04-16T07:38:27.825 INFO:teuthology.orchestra.run.smithi079.stdout:(6/11): libcephsqlite-19.0.0-3109.g30b0aa00.el9 185 kB/s | 173 kB 00:00 2024-04-16T07:38:27.992 INFO:teuthology.orchestra.run.smithi079.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 362 kB/s | 187 kB 00:00 2024-04-16T07:38:28.101 INFO:teuthology.orchestra.run.smithi079.stdout:(8/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 1.7 MB/s | 469 kB 00:00 2024-04-16T07:38:28.235 INFO:teuthology.orchestra.run.smithi079.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 772 kB/s | 187 kB 00:00 2024-04-16T07:38:28.327 INFO:teuthology.orchestra.run.smithi079.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 543 kB/s | 122 kB 00:00 2024-04-16T07:39:52.753 INFO:teuthology.orchestra.run.smithi079.stdout:(11/11): ceph-test-19.0.0-3109.g30b0aa00.el9.x8 1.0 MB/s | 84 MB 01:25 2024-04-16T07:39:52.756 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:39:52.756 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.0 MB/s | 87 MB 01:26 2024-04-16T07:39:52.791 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:39:52.814 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:39:52.814 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:39:53.468 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:39:53.468 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:39:54.479 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:39:54.544 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T07:39:54.587 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T07:39:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:39:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:39:54.717 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T07:39:54.757 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T07:39:54.802 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T07:39:54.833 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T07:39:54.879 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T07:39:54.917 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T07:39:54.944 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:40:06.426 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:40:06.502 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:40:13.094 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:40:13.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/11 2024-04-16T07:40:13.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/11 2024-04-16T07:40:13.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T07:40:13.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T07:40:13.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T07:40:13.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T07:40:13.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T07:40:13.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T07:40:13.097 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T07:40:13.097 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T07:40:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T07:40:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:40:13.477 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:40:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:40:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T07:40:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T07:40:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T07:40:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:40:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T07:40:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:40:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:40:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T07:40:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:40:13.479 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:40:13.748 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2024-04-16T07:40:14.334 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:32 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:40:14.487 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:40:14.490 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================= 2024-04-16T07:40:14.490 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:40:14.490 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================= 2024-04-16T07:40:14.490 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:40:14.490 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 6.5 k 2024-04-16T07:40:14.490 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 2.2 M 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 1.6 M 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 249 k 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.0 M 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 71 M 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:40:14.491 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:40:14.492 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================= 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout:Install 36 Packages 2024-04-16T07:40:14.493 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:40:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 84 M 2024-04-16T07:40:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 922 M 2024-04-16T07:40:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:40:15.656 INFO:teuthology.orchestra.run.smithi079.stdout:(1/36): ceph-19.0.0-3109.g30b0aa00.el9.x86_64.r 17 kB/s | 6.5 kB 00:00 2024-04-16T07:40:17.273 INFO:teuthology.orchestra.run.smithi079.stdout:(2/36): ceph-mgr-19.0.0-3109.g30b0aa00.el9.x86_ 796 kB/s | 1.6 MB 00:02 2024-04-16T07:40:17.498 INFO:teuthology.orchestra.run.smithi079.stdout:(3/36): ceph-mds-19.0.0-3109.g30b0aa00.el9.x86_ 1.0 MB/s | 2.2 MB 00:02 2024-04-16T07:40:17.748 INFO:teuthology.orchestra.run.smithi079.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3109.g30b0 994 kB/s | 249 kB 00:00 2024-04-16T07:40:17.882 INFO:teuthology.orchestra.run.smithi079.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-04-16T07:40:17.924 INFO:teuthology.orchestra.run.smithi079.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-16T07:40:17.957 INFO:teuthology.orchestra.run.smithi079.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 941 kB/s | 32 kB 00:00 2024-04-16T07:40:17.991 INFO:teuthology.orchestra.run.smithi079.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T07:40:18.025 INFO:teuthology.orchestra.run.smithi079.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T07:40:18.067 INFO:teuthology.orchestra.run.smithi079.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-04-16T07:40:18.142 INFO:teuthology.orchestra.run.smithi079.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-04-16T07:40:18.176 INFO:teuthology.orchestra.run.smithi079.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-04-16T07:40:18.209 INFO:teuthology.orchestra.run.smithi079.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-04-16T07:40:18.243 INFO:teuthology.orchestra.run.smithi079.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-04-16T07:40:18.785 INFO:teuthology.orchestra.run.smithi079.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 230 kB/s | 125 kB 00:00 2024-04-16T07:40:18.910 INFO:teuthology.orchestra.run.smithi079.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-04-16T07:40:19.351 INFO:teuthology.orchestra.run.smithi167.stdout:(11/11): ceph-test-19.0.0-3109.g30b0aa00.el9.x8 745 kB/s | 84 MB 01:55 2024-04-16T07:40:19.353 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:40:19.353 INFO:teuthology.orchestra.run.smithi167.stdout:Total 764 kB/s | 87 MB 01:56 2024-04-16T07:40:19.388 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:40:19.411 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:40:19.411 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:40:19.511 INFO:teuthology.orchestra.run.smithi079.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 252 kB/s | 151 kB 00:00 2024-04-16T07:40:19.653 INFO:teuthology.orchestra.run.smithi079.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 307 kB/s | 43 kB 00:00 2024-04-16T07:40:19.828 INFO:teuthology.orchestra.run.smithi079.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 979 kB/s | 172 kB 00:00 2024-04-16T07:40:20.031 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:40:20.032 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:40:20.053 INFO:teuthology.orchestra.run.smithi079.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-04-16T07:40:20.162 INFO:teuthology.orchestra.run.smithi079.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-04-16T07:40:20.271 INFO:teuthology.orchestra.run.smithi079.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-04-16T07:40:20.380 INFO:teuthology.orchestra.run.smithi079.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 214 kB/s | 23 kB 00:00 2024-04-16T07:40:20.489 INFO:teuthology.orchestra.run.smithi079.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-04-16T07:40:20.605 INFO:teuthology.orchestra.run.smithi079.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 169 kB/s | 20 kB 00:00 2024-04-16T07:40:20.714 INFO:teuthology.orchestra.run.smithi079.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-04-16T07:40:20.831 INFO:teuthology.orchestra.run.smithi079.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 677 kB/s | 79 kB 00:00 2024-04-16T07:40:20.973 INFO:teuthology.orchestra.run.smithi079.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-04-16T07:40:21.082 INFO:teuthology.orchestra.run.smithi079.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-04-16T07:40:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:40:21.161 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T07:40:21.199 INFO:teuthology.orchestra.run.smithi079.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 770 kB/s | 90 kB 00:00 2024-04-16T07:40:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T07:40:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:40:21.305 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:40:21.307 INFO:teuthology.orchestra.run.smithi079.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2024-04-16T07:40:21.337 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T07:40:21.381 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T07:40:21.427 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T07:40:21.432 INFO:teuthology.orchestra.run.smithi079.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-04-16T07:40:21.457 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T07:40:21.496 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T07:40:21.541 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T07:40:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:40:21.591 INFO:teuthology.orchestra.run.smithi079.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-04-16T07:40:21.700 INFO:teuthology.orchestra.run.smithi079.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-04-16T07:40:24.700 INFO:teuthology.orchestra.run.smithi079.stdout:(35/36): ceph-mon-19.0.0-3109.g30b0aa00.el9.x86 569 kB/s | 5.0 MB 00:09 2024-04-16T07:40:33.139 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:40:33.218 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:40:39.626 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:40:39.626 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/11 2024-04-16T07:40:39.626 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/11 2024-04-16T07:40:39.626 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T07:40:39.626 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T07:40:39.626 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T07:40:39.627 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T07:40:39.627 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T07:40:39.627 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T07:40:39.627 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T07:40:39.627 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T07:40:40.012 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T07:40:40.012 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:40:40.012 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:40:40.013 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:40.013 INFO:teuthology.orchestra.run.smithi167.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:40:40.013 INFO:teuthology.orchestra.run.smithi167.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T07:40:40.013 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:40:40.013 INFO:teuthology.orchestra.run.smithi167.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T07:40:40.014 INFO:teuthology.orchestra.run.smithi167.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T07:40:40.014 INFO:teuthology.orchestra.run.smithi167.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:40:40.014 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T07:40:40.014 INFO:teuthology.orchestra.run.smithi167.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:40:40.014 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:40:40.015 INFO:teuthology.orchestra.run.smithi167.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T07:40:40.015 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:40:40.015 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:40:40.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph 2024-04-16T07:40:40.896 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:03:02 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:40:41.050 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:40:41.052 INFO:teuthology.orchestra.run.smithi167.stdout:========================================================================================= 2024-04-16T07:40:41.052 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout:========================================================================================= 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: ceph x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 6.5 k 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 2.2 M 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 1.6 M 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 249 k 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.0 M 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 71 M 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:40:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:40:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:40:41.055 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:40:41.056 INFO:teuthology.orchestra.run.smithi167.stdout:========================================================================================= 2024-04-16T07:40:41.056 INFO:teuthology.orchestra.run.smithi167.stdout:Install 36 Packages 2024-04-16T07:40:41.056 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:40:41.056 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 84 M 2024-04-16T07:40:41.056 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 922 M 2024-04-16T07:40:41.056 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:40:41.911 INFO:teuthology.orchestra.run.smithi167.stdout:(1/36): ceph-19.0.0-3109.g30b0aa00.el9.x86_64.r 17 kB/s | 6.5 kB 00:00 2024-04-16T07:40:45.252 INFO:teuthology.orchestra.run.smithi167.stdout:(2/36): ceph-mon-19.0.0-3109.g30b0aa00.el9.x86_ 1.5 MB/s | 5.0 MB 00:03 2024-04-16T07:40:45.794 INFO:teuthology.orchestra.run.smithi167.stdout:(3/36): ceph-mgr-19.0.0-3109.g30b0aa00.el9.x86_ 374 kB/s | 1.6 MB 00:04 2024-04-16T07:40:46.236 INFO:teuthology.orchestra.run.smithi167.stdout:(4/36): ceph-mds-19.0.0-3109.g30b0aa00.el9.x86_ 476 kB/s | 2.2 MB 00:04 2024-04-16T07:40:46.361 INFO:teuthology.orchestra.run.smithi167.stdout:(5/36): ceph-mgr-modules-core-19.0.0-3109.g30b0 439 kB/s | 249 kB 00:00 2024-04-16T07:40:46.487 INFO:teuthology.orchestra.run.smithi167.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 747 kB/s | 187 kB 00:00 2024-04-16T07:40:46.512 INFO:teuthology.orchestra.run.smithi167.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 961 kB/s | 144 kB 00:00 2024-04-16T07:40:46.529 INFO:teuthology.orchestra.run.smithi167.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 759 kB/s | 32 kB 00:00 2024-04-16T07:40:46.546 INFO:teuthology.orchestra.run.smithi167.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T07:40:46.563 INFO:teuthology.orchestra.run.smithi167.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T07:40:46.613 INFO:teuthology.orchestra.run.smithi167.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-04-16T07:40:46.688 INFO:teuthology.orchestra.run.smithi167.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 9.1 MB/s | 1.1 MB 00:00 2024-04-16T07:40:46.705 INFO:teuthology.orchestra.run.smithi167.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-04-16T07:40:46.722 INFO:teuthology.orchestra.run.smithi167.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-04-16T07:40:46.748 INFO:teuthology.orchestra.run.smithi167.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-04-16T07:40:46.898 INFO:teuthology.orchestra.run.smithi167.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 714 kB/s | 125 kB 00:00 2024-04-16T07:40:46.998 INFO:teuthology.orchestra.run.smithi167.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-16T07:40:47.090 INFO:teuthology.orchestra.run.smithi167.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 661 kB/s | 43 kB 00:00 2024-04-16T07:40:47.174 INFO:teuthology.orchestra.run.smithi167.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-16T07:40:47.299 INFO:teuthology.orchestra.run.smithi167.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-04-16T07:40:47.341 INFO:teuthology.orchestra.run.smithi167.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-04-16T07:40:47.383 INFO:teuthology.orchestra.run.smithi167.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-04-16T07:40:47.416 INFO:teuthology.orchestra.run.smithi167.stdout:(23/36): lua-devel-5.4.4-4.el9.x86_64.rpm 33 kB/s | 22 kB 00:00 2024-04-16T07:40:47.442 INFO:teuthology.orchestra.run.smithi167.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-04-16T07:40:47.484 INFO:teuthology.orchestra.run.smithi167.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 473 kB/s | 20 kB 00:00 2024-04-16T07:40:47.509 INFO:teuthology.orchestra.run.smithi167.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-04-16T07:40:47.535 INFO:teuthology.orchestra.run.smithi167.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-04-16T07:40:47.602 INFO:teuthology.orchestra.run.smithi167.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 855 kB/s | 79 kB 00:00 2024-04-16T07:40:47.660 INFO:teuthology.orchestra.run.smithi167.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-04-16T07:40:47.686 INFO:teuthology.orchestra.run.smithi167.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-04-16T07:40:47.736 INFO:teuthology.orchestra.run.smithi167.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-16T07:40:47.761 INFO:teuthology.orchestra.run.smithi167.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 476 kB/s | 36 kB 00:00 2024-04-16T07:40:47.845 INFO:teuthology.orchestra.run.smithi167.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-16T07:40:47.895 INFO:teuthology.orchestra.run.smithi167.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 402 kB/s | 20 kB 00:00 2024-04-16T07:40:48.029 INFO:teuthology.orchestra.run.smithi167.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-04-16T07:41:34.604 INFO:teuthology.orchestra.run.smithi079.stdout:(36/36): ceph-osd-19.0.0-3109.g30b0aa00.el9.x86 939 kB/s | 71 MB 01:17 2024-04-16T07:41:34.607 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:41:34.607 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.1 MB/s | 84 MB 01:20 2024-04-16T07:41:34.628 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:41:34.667 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:41:34.667 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:41:35.400 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:41:35.400 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:41:36.997 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:41:37.055 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T07:41:37.097 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T07:41:37.152 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T07:41:37.228 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T07:41:37.266 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T07:41:37.298 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T07:41:37.390 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T07:41:37.477 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T07:41:37.518 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T07:41:37.589 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T07:41:37.644 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T07:41:37.682 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T07:41:37.731 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T07:41:37.786 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T07:41:37.980 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T07:41:38.029 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T07:41:38.060 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T07:41:38.094 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T07:41:38.124 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T07:41:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T07:41:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:41:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T07:41:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T07:41:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T07:41:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T07:41:38.468 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T07:41:38.519 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T07:41:38.590 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T07:41:38.727 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T07:41:38.845 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:41:39.039 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 31/36 2024-04-16T07:41:39.074 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:41:39.100 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:41:39.100 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:39.100 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:41:39.101 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:41:39.101 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:41:39.101 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:41:49.898 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:41:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:41:49.926 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:49.926 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:41:49.926 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:41:49.926 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:41:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:41:50.466 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:41:50.490 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:41:50.490 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:50.490 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:41:50.490 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:41:50.490 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:41:50.490 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:41:50.725 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:41:50.749 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:41:50.749 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:41:50.749 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:41:50.749 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:41:50.749 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:41:50.749 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:41:50.822 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:41:57.773 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:41:57.773 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/36 2024-04-16T07:41:57.774 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/36 2024-04-16T07:41:57.774 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/36 2024-04-16T07:41:57.774 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/36 2024-04-16T07:41:57.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/36 2024-04-16T07:41:57.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 6/36 2024-04-16T07:41:57.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T07:41:57.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T07:41:57.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T07:41:57.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T07:41:57.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T07:41:57.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T07:41:57.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T07:41:57.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T07:41:57.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T07:41:57.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T07:41:57.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T07:41:57.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T07:41:57.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T07:41:57.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T07:41:57.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:41:57.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T07:41:57.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T07:41:57.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T07:41:57.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T07:41:57.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T07:41:57.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T07:41:57.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T07:41:57.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T07:41:57.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:41:57.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T07:41:57.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T07:41:57.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T07:41:57.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T07:41:57.780 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T07:41:58.697 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T07:41:58.697 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:41:58.697 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:41:58.697 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:58.697 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:58.697 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:58.697 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:41:58.697 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:58.697 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:41:58.698 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:41:58.699 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:41:58.700 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:41:58.700 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:41:58.700 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:41:58.700 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:41:58.700 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:41:58.961 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2024-04-16T07:41:59.539 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:17 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:41:59.635 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:41:59.688 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:41:59.689 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-16T07:41:59.689 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:41:59.758 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2024-04-16T07:42:00.339 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:18 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:42:00.490 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:42:00.712 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:42:00.712 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:42:00.712 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:42:00.712 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:42:00.712 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 762 k 2024-04-16T07:42:00.712 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:00.712 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:42:00.713 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:42:00.713 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-16T07:42:00.713 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:00.713 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 762 k 2024-04-16T07:42:00.713 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 769 k 2024-04-16T07:42:00.713 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:42:01.995 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-19.0.0-3109.g30b0aa00.el9.noarch.rpm 507 kB/s | 762 kB 00:01 2024-04-16T07:42:01.996 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:01.996 INFO:teuthology.orchestra.run.smithi079.stdout:Total 507 kB/s | 762 kB 00:01 2024-04-16T07:42:01.996 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:42:02.003 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:42:02.003 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:42:02.009 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:42:02.010 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:42:02.084 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:42:02.553 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:42:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:42:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:42:03.328 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:42:03.328 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:03.328 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:42:03.328 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:03.328 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:03.328 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:42:03.510 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:42:04.100 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:22 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:42:04.254 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:42:04.255 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:42:04.255 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-04-16T07:42:04.255 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:42:04.255 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:42:04.256 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 157 k 2024-04-16T07:42:04.256 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:04.256 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:42:04.256 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:42:04.256 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-16T07:42:04.256 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:04.256 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 157 k 2024-04-16T07:42:04.256 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 640 k 2024-04-16T07:42:04.256 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:42:05.553 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-19.0.0-3109.g30b0aa 121 kB/s | 157 kB 00:01 2024-04-16T07:42:05.554 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:05.554 INFO:teuthology.orchestra.run.smithi079.stdout:Total 121 kB/s | 157 kB 00:01 2024-04-16T07:42:05.554 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:42:05.565 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:42:05.565 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:42:05.621 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:42:05.621 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:42:05.783 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:42:05.854 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:42:05.880 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:42:05.880 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:42:05.880 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:42:05.880 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:06.852 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:42:06.853 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:06.853 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:42:06.853 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:42:06.853 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:06.853 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:42:07.117 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2024-04-16T07:42:07.706 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:25 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:42:07.804 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:42:07.856 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:42:07.857 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-16T07:42:07.857 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:42:07.923 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:42:08.500 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:26 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:42:08.651 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:42:08.652 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 3.5 M 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 34 k 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 16 k 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T07:42:08.653 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T07:42:08.654 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T07:42:08.654 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:42:08.654 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:42:08.654 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:08.654 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:42:08.654 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-04-16T07:42:08.654 INFO:teuthology.orchestra.run.smithi079.stdout:Install 11 Packages 2024-04-16T07:42:08.654 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:08.655 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 7.6 M 2024-04-16T07:42:08.655 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 100 M 2024-04-16T07:42:08.655 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:42:09.259 INFO:teuthology.orchestra.run.smithi079.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3109.g30b 36 kB/s | 16 kB 00:00 2024-04-16T07:42:09.276 INFO:teuthology.orchestra.run.smithi079.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3109.g30 72 kB/s | 34 kB 00:00 2024-04-16T07:42:09.410 INFO:teuthology.orchestra.run.smithi079.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-04-16T07:42:09.518 INFO:teuthology.orchestra.run.smithi079.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-04-16T07:42:09.760 INFO:teuthology.orchestra.run.smithi079.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 81 kB/s | 19 kB 00:00 2024-04-16T07:42:09.844 INFO:teuthology.orchestra.run.smithi079.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-04-16T07:42:10.027 INFO:teuthology.orchestra.run.smithi079.stdout:(7/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.6 MB/s | 2.0 MB 00:00 2024-04-16T07:42:10.053 INFO:teuthology.orchestra.run.smithi079.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 694 kB/s | 144 kB 00:00 2024-04-16T07:42:10.103 INFO:teuthology.orchestra.run.smithi079.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 616 kB/s | 31 kB 00:00 2024-04-16T07:42:10.128 INFO:teuthology.orchestra.run.smithi079.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-04-16T07:42:16.294 INFO:teuthology.orchestra.run.smithi079.stdout:(11/11): ceph-mgr-dashboard-19.0.0-3109.g30b0aa 484 kB/s | 3.5 MB 00:07 2024-04-16T07:42:16.298 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:16.299 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.0 MB/s | 7.6 MB 00:07 2024-04-16T07:42:16.319 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:42:16.339 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:42:16.339 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:42:16.513 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:42:16.513 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:42:16.991 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:42:17.067 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T07:42:17.097 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T07:42:17.224 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T07:42:17.348 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T07:42:17.410 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T07:42:17.493 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T07:42:17.545 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T07:42:17.574 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:42:17.607 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 9/11 2024-04-16T07:42:18.649 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 10/11 2024-04-16T07:42:18.698 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:42:20.052 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:42:20.052 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 1/11 2024-04-16T07:42:20.052 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 2/11 2024-04-16T07:42:20.052 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 3/11 2024-04-16T07:42:20.052 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T07:42:20.052 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T07:42:20.052 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T07:42:20.052 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T07:42:20.053 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:42:20.053 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T07:42:20.053 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T07:42:20.391 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T07:42:20.392 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T07:42:20.393 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T07:42:20.393 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:42:20.393 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:42:20.393 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:20.393 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:42:20.646 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:42:21.230 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:39 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:42:21.382 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:42:21.383 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-04-16T07:42:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:42:21.383 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-04-16T07:42:21.383 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 7.4 M 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:42:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:21.385 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:42:21.385 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-04-16T07:42:21.385 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-04-16T07:42:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:21.385 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 37 M 2024-04-16T07:42:21.385 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 209 M 2024-04-16T07:42:21.385 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:42:21.738 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 251 kB/s | 32 kB 00:00 2024-04-16T07:42:21.788 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 341 kB/s | 17 kB 00:00 2024-04-16T07:42:21.838 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 694 kB/s | 34 kB 00:00 2024-04-16T07:42:22.380 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.9 MB/s | 3.0 MB 00:00 2024-04-16T07:42:23.014 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.0 MB/s | 4.7 MB 00:01 2024-04-16T07:42:23.314 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.1 MB/s | 4.8 MB 00:00 2024-04-16T07:42:23.339 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 670 kB/s | 217 kB 00:00 2024-04-16T07:42:24.098 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-04-16T07:42:24.349 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-04-16T07:42:24.374 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 712 kB/s | 195 kB 00:00 2024-04-16T07:42:24.699 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 701 kB/s | 245 kB 00:00 2024-04-16T07:42:37.431 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 479 kB/s | 7.4 MB 00:15 2024-04-16T07:42:37.434 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:37.435 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.3 MB/s | 37 MB 00:16 2024-04-16T07:42:37.479 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:42:37.505 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:42:37.505 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:42:37.870 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:42:37.870 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:42:38.575 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:42:38.653 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:42:38.684 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:42:38.778 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:42:38.821 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:42:38.999 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:42:39.028 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:42:39.106 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:42:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:42:39.869 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:42:41.085 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:42:42.110 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:42:42.141 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 1/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:42:44.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:42:44.721 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:42:44.721 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:42:45.135 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:42:45.136 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:45.136 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:42:45.136 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:45.136 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:42:45.136 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:42:45.136 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:42:45.137 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:42:45.137 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:42:45.137 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:42:45.137 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:42:45.137 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:42:45.137 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:42:45.137 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:42:45.138 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:42:45.138 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:45.138 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:42:45.573 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:42:46.161 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:04 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:42:46.314 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:42:46.315 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-04-16T07:42:46.315 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 49 k 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:42:46.316 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:42:46.317 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:42:46.317 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:42:46.317 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:42:46.317 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:42:46.317 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:46.317 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:42:46.317 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-04-16T07:42:46.317 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2024-04-16T07:42:46.317 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:46.318 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.0 M 2024-04-16T07:42:46.318 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-04-16T07:42:46.318 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:42:46.852 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 186 kB/s | 24 kB 00:00 2024-04-16T07:42:46.877 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-04-16T07:42:46.936 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2024-04-16T07:42:46.961 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-04-16T07:42:46.987 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T07:42:47.012 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-04-16T07:42:47.146 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-04-16T07:42:47.171 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-04-16T07:42:47.330 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): ceph-mgr-rook-19.0.0-3109.g30b0aa00.el9 81 kB/s | 49 kB 00:00 2024-04-16T07:42:47.355 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 969 kB/s | 203 kB 00:00 2024-04-16T07:42:47.422 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-04-16T07:42:47.481 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-04-16T07:42:47.514 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 321 kB/s | 59 kB 00:00 2024-04-16T07:42:47.516 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:47.516 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-04-16T07:42:47.517 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:42:47.530 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:42:47.530 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:42:47.621 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:42:47.623 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:42:48.127 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:42:48.295 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:42:48.349 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:42:48.404 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:42:48.442 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:42:48.483 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:42:48.577 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:42:48.678 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:42:48.727 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:42:49.450 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:42:49.508 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:42:49.542 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:42:49.590 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:42:49.612 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:42:50.380 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 1/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:42:50.381 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:42:50.745 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:42:50.745 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:50.745 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:42:50.745 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:50.745 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:42:50.745 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:50.746 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:42:50.994 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:42:51.612 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:09 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:42:51.768 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:42:51.768 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 142 k 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:51.769 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:42:51.770 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-04-16T07:42:51.770 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-04-16T07:42:51.770 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:51.770 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 6.8 M 2024-04-16T07:42:51.770 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 33 M 2024-04-16T07:42:51.770 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:42:52.245 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-04-16T07:42:52.272 INFO:teuthology.orchestra.run.smithi167.stdout:(36/36): ceph-osd-19.0.0-3109.g30b0aa00.el9.x86 571 kB/s | 71 MB 02:07 2024-04-16T07:42:52.275 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:52.276 INFO:teuthology.orchestra.run.smithi167.stdout:Total 657 kB/s | 84 MB 02:11 2024-04-16T07:42:52.295 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:42:52.335 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:42:52.336 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:42:52.562 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-04-16T07:42:52.587 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-04-16T07:42:52.646 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 870 kB/s | 51 kB 00:00 2024-04-16T07:42:52.738 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 329 kB/s | 58 kB 00:00 2024-04-16T07:42:53.088 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): ceph-mgr-cephadm-19.0.0-3109.g30b0aa00.e 143 kB/s | 142 kB 00:00 2024-04-16T07:42:53.089 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:42:53.089 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:42:53.090 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:53.090 INFO:teuthology.orchestra.run.smithi079.stdout:Total 5.1 MB/s | 6.8 MB 00:01 2024-04-16T07:42:53.103 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:42:53.114 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:42:53.114 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:42:53.203 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:42:53.203 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:42:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:42:53.669 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:42:53.714 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:42:54.229 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:42:54.309 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:42:54.372 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:42:54.402 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:42:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:42:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T07:42:54.785 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T07:42:54.819 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T07:42:54.889 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T07:42:54.926 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T07:42:54.959 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T07:42:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T07:42:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/6 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:42:55.183 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:42:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T07:42:55.259 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T07:42:55.313 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T07:42:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T07:42:55.399 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T07:42:55.455 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T07:42:55.542 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:42:55.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:55.542 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:42:55.543 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:42:55.543 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:42:55.543 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:42:55.543 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:42:55.543 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:42:55.543 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:42:55.543 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:55.543 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:42:55.677 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T07:42:55.723 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T07:42:55.767 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T07:42:55.786 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2024-04-16T07:42:55.804 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T07:42:55.834 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T07:42:55.910 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T07:42:55.944 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:42:55.978 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T07:42:56.018 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T07:42:56.053 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T07:42:56.087 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T07:42:56.176 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T07:42:56.210 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T07:42:56.283 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T07:42:56.367 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:14 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:42:56.428 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T07:42:56.520 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:42:56.521 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:42:56.521 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:42:56.521 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:42:56.521 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:42:56.521 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 871 k 2024-04-16T07:42:56.521 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:42:56.522 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:42:56.522 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:56.522 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:42:56.522 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:42:56.522 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-04-16T07:42:56.522 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:56.523 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 950 k 2024-04-16T07:42:56.523 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 4.0 M 2024-04-16T07:42:56.523 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:42:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:42:56.657 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 584 kB/s | 79 kB 00:00 2024-04-16T07:42:56.741 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 31/36 2024-04-16T07:42:56.765 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:42:56.791 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:42:56.791 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:42:56.791 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:42:56.791 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:42:56.791 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:42:56.791 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:42:58.449 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-fuse-19.0.0-3109.g30b0aa00.el9.x86_ 452 kB/s | 871 kB 00:01 2024-04-16T07:42:58.451 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:42:58.451 INFO:teuthology.orchestra.run.smithi079.stdout:Total 492 kB/s | 950 kB 00:01 2024-04-16T07:42:58.451 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:42:58.478 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:42:58.478 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:42:58.555 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:42:58.555 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:42:58.746 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:42:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:42:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:42:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:42:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/2 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:42:59.975 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:00.270 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-volume 2024-04-16T07:43:00.865 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:18 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:01.018 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:01.018 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 273 k 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-04-16T07:43:01.019 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:01.020 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 342 k 2024-04-16T07:43:01.020 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 1.5 M 2024-04-16T07:43:01.020 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:43:01.139 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 577 kB/s | 69 kB 00:00 2024-04-16T07:43:03.448 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-volume-19.0.0-3109.g30b0aa00.el9.no 112 kB/s | 273 kB 00:02 2024-04-16T07:43:03.450 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:03.450 INFO:teuthology.orchestra.run.smithi079.stdout:Total 141 kB/s | 342 kB 00:02 2024-04-16T07:43:03.450 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:43:03.460 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:43:03.460 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:43:03.508 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:43:03.509 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:43:03.735 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:43:03.920 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T07:43:03.959 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:43:03.976 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:43:03.976 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:03.976 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T07:43:03.976 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:04.675 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 1/2 2024-04-16T07:43:05.007 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T07:43:05.007 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:05.007 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:43:05.007 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:05.007 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T07:43:05.007 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:05.007 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:05.309 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2024-04-16T07:43:05.901 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:23 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:06.052 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:06.052 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 125 k 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 125 k 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 445 k 2024-04-16T07:43:06.053 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:43:06.697 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-19.0.0-3109.g30b0aa00.el9.x86_64 194 kB/s | 125 kB 00:00 2024-04-16T07:43:06.698 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:06.698 INFO:teuthology.orchestra.run.smithi079.stdout:Total 194 kB/s | 125 kB 00:00 2024-04-16T07:43:06.698 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:43:06.703 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:43:06.703 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:43:06.759 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:43:06.760 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:43:06.899 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:43:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:07.417 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:07.695 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:07.695 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:07.696 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:43:07.696 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:07.696 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:07.696 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:07.937 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2024-04-16T07:43:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:43:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:43:08.163 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:08.163 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:43:08.163 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:43:08.163 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:43:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:08.527 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:26 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:08.630 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:08.684 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:08.685 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-16T07:43:08.685 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:43:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:43:08.739 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:08.739 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:43:08.739 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:43:08.740 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:43:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:08.760 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2024-04-16T07:43:08.984 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:43:09.008 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:43:09.008 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:09.008 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:43:09.008 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:43:09.008 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:43:09.008 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:09.065 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:43:09.344 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:27 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:09.495 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:09.495 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:09.495 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-04-16T07:43:09.495 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 32 k 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 32 k 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 142 k 2024-04-16T07:43:09.496 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:43:09.961 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-19.0.0-3109.g30b0aa00.el9.x86_6 68 kB/s | 32 kB 00:00 2024-04-16T07:43:09.961 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:09.962 INFO:teuthology.orchestra.run.smithi079.stdout:Total 68 kB/s | 32 kB 00:00 2024-04-16T07:43:09.962 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:43:09.966 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:43:09.966 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:43:09.989 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:43:09.989 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:43:10.076 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:43:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:10.822 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:43:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:10.822 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:11.031 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2024-04-16T07:43:11.614 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:29 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:11.714 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:11.766 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:11.767 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-16T07:43:11.767 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:11.846 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2024-04-16T07:43:12.428 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:30 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:12.528 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:12.580 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:12.581 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-16T07:43:12.581 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:12.670 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2024-04-16T07:43:13.252 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:31 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:13.352 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:13.403 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:13.404 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-16T07:43:13.405 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:13.487 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2024-04-16T07:43:14.066 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:32 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:14.166 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:14.219 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:14.220 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-16T07:43:14.220 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:14.300 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2024-04-16T07:43:14.878 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:32 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:14.976 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:15.028 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:15.029 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-16T07:43:15.029 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:15.111 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2024-04-16T07:43:15.700 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:33 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:15.800 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:15.851 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:15.852 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-16T07:43:15.852 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:15.933 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2024-04-16T07:43:16.172 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 6/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T07:43:16.173 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T07:43:16.174 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T07:43:16.175 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T07:43:16.175 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T07:43:16.175 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T07:43:16.175 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:43:16.175 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T07:43:16.175 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T07:43:16.175 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T07:43:16.175 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T07:43:16.175 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T07:43:16.520 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:34 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:16.672 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 90 k 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:16.673 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 90 k 2024-04-16T07:43:16.674 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 313 k 2024-04-16T07:43:16.674 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:43:17.029 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T07:43:17.029 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:17.029 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:43:17.029 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:17.029 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:17.029 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:17.029 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:17.029 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:17.029 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:43:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:43:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:43:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:43:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:43:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:43:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:17.032 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:43:17.281 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 148 kB/s | 90 kB 00:00 2024-04-16T07:43:17.281 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:17.281 INFO:teuthology.orchestra.run.smithi079.stdout:Total 147 kB/s | 90 kB 00:00 2024-04-16T07:43:17.282 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:43:17.290 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:43:17.291 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:43:17.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base 2024-04-16T07:43:17.347 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:43:17.348 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:43:17.493 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:43:17.595 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:17.919 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:39 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:43:17.969 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:18.019 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:18.071 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:43:18.072 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-16T07:43:18.072 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:43:18.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install cephadm 2024-04-16T07:43:18.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:18.262 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:18.262 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:43:18.262 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:18.262 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:18.262 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:18.542 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2024-04-16T07:43:18.712 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:40 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:43:18.863 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:43:18.863 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 762 k 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 762 k 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 769 k 2024-04-16T07:43:18.864 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:43:19.134 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:37 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:19.291 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:19.292 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:19.292 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-04-16T07:43:19.292 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:19.292 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:43:19.292 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:43:19.292 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:19.292 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:43:19.292 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:19.292 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-16T07:43:19.293 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:19.293 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.4 M 2024-04-16T07:43:19.293 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 29 M 2024-04-16T07:43:19.293 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:43:20.626 INFO:teuthology.orchestra.run.smithi167.stdout:cephadm-19.0.0-3109.g30b0aa00.el9.noarch.rpm 433 kB/s | 762 kB 00:01 2024-04-16T07:43:20.627 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:20.627 INFO:teuthology.orchestra.run.smithi167.stdout:Total 432 kB/s | 762 kB 00:01 2024-04-16T07:43:20.627 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:43:20.634 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:43:20.634 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:43:20.640 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:43:20.640 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:43:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:43:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:43:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:43:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:43:21.924 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:43:21.925 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:21.925 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:43:21.925 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:21.925 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:21.925 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:43:22.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:43:22.731 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:44 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:43:22.889 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:43:22.890 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:43:22.890 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-16T07:43:22.890 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:43:22.890 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:43:22.890 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 157 k 2024-04-16T07:43:22.890 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:22.890 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:43:22.890 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:43:22.891 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-16T07:43:22.891 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:22.891 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 157 k 2024-04-16T07:43:22.891 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 640 k 2024-04-16T07:43:22.891 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:43:24.067 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-immutable-object-cache-19.0.0-3109.g30b0aa 134 kB/s | 157 kB 00:01 2024-04-16T07:43:24.067 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:24.068 INFO:teuthology.orchestra.run.smithi167.stdout:Total 134 kB/s | 157 kB 00:01 2024-04-16T07:43:24.068 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:43:24.079 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:43:24.079 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:43:24.136 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:43:24.137 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:43:24.354 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:43:24.434 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:43:24.459 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:43:24.459 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:24.459 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:43:24.459 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:25.432 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:43:25.432 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:25.432 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:43:25.432 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:25.432 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:25.432 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:43:25.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr 2024-04-16T07:43:26.306 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:48 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:43:26.405 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:43:26.458 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:43:26.459 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-16T07:43:26.459 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:43:26.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:43:27.139 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:49 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:43:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:43:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2024-04-16T07:43:27.300 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:43:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2024-04-16T07:43:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 3.5 M 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 34 k 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 16 k 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:43:27.301 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:27.302 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:43:27.302 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2024-04-16T07:43:27.302 INFO:teuthology.orchestra.run.smithi167.stdout:Install 11 Packages 2024-04-16T07:43:27.302 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:27.302 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 7.6 M 2024-04-16T07:43:27.302 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 100 M 2024-04-16T07:43:27.302 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:43:27.997 INFO:teuthology.orchestra.run.smithi167.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3109.g30b 37 kB/s | 16 kB 00:00 2024-04-16T07:43:28.214 INFO:teuthology.orchestra.run.smithi167.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3109.g30 51 kB/s | 34 kB 00:00 2024-04-16T07:43:28.239 INFO:teuthology.orchestra.run.smithi167.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 983 kB/s | 238 kB 00:00 2024-04-16T07:43:28.523 INFO:teuthology.orchestra.run.smithi167.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-04-16T07:43:28.765 INFO:teuthology.orchestra.run.smithi167.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-04-16T07:43:28.840 INFO:teuthology.orchestra.run.smithi167.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 912 kB/s | 548 kB 00:00 2024-04-16T07:43:28.949 INFO:teuthology.orchestra.run.smithi167.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-04-16T07:43:29.057 INFO:teuthology.orchestra.run.smithi167.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-04-16T07:43:29.183 INFO:teuthology.orchestra.run.smithi167.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-04-16T07:43:29.360 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 346 kB/s | 3.4 MB 00:10 2024-04-16T07:43:29.360 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:29.361 INFO:teuthology.orchestra.run.smithi079.stdout:Total 346 kB/s | 3.4 MB 00:10 2024-04-16T07:43:29.361 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:43:29.374 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:43:29.374 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:43:29.392 INFO:teuthology.orchestra.run.smithi167.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-04-16T07:43:29.452 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:43:29.453 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:43:30.008 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:43:30.069 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:30.095 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:30.095 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:30.095 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:43:30.095 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:43:30.096 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:43:30.096 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:31.295 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:43:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:31.295 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:31.534 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2024-04-16T07:43:32.138 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:05:50 ago on Tue 16 Apr 2024 07:37:42 AM UTC. 2024-04-16T07:43:32.292 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 175 k 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 175 k 2024-04-16T07:43:32.293 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 673 k 2024-04-16T07:43:32.294 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-16T07:43:33.565 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 138 kB/s | 175 kB 00:01 2024-04-16T07:43:33.566 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:33.566 INFO:teuthology.orchestra.run.smithi079.stdout:Total 138 kB/s | 175 kB 00:01 2024-04-16T07:43:33.566 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-16T07:43:33.578 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-16T07:43:33.578 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-16T07:43:33.639 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-16T07:43:33.640 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-16T07:43:33.795 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-16T07:43:33.898 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:34.335 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:34.616 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:43:34.616 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:34.617 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-16T07:43:34.617 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:34.617 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:43:34.617 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-16T07:43:34.896 DEBUG:teuthology.parallel:result is None 2024-04-16T07:43:39.499 INFO:teuthology.orchestra.run.smithi167.stdout:(11/11): ceph-mgr-dashboard-19.0.0-3109.g30b0aa 303 kB/s | 3.5 MB 00:11 2024-04-16T07:43:39.503 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:39.503 INFO:teuthology.orchestra.run.smithi167.stdout:Total 639 kB/s | 7.6 MB 00:12 2024-04-16T07:43:39.523 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:43:39.542 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:43:39.542 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:43:39.729 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:43:39.729 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:43:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:43:40.413 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T07:43:40.453 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T07:43:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T07:43:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T07:43:40.731 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T07:43:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T07:43:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T07:43:40.905 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:43:40.936 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 9/11 2024-04-16T07:43:41.981 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 10/11 2024-04-16T07:43:42.020 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 1/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 2/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 3/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:43:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T07:43:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T07:43:43.702 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T07:43:43.703 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:43:43.703 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:43:43.703 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:43.703 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:43:43.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:43:44.561 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:06 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:43:44.717 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:43:44.718 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================================ 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================================ 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 7.4 M 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:43:44.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:43:44.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:43:44.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:43:44.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:43:44.720 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:44.720 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:43:44.720 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================================ 2024-04-16T07:43:44.720 INFO:teuthology.orchestra.run.smithi167.stdout:Install 12 Packages 2024-04-16T07:43:44.720 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:43:44.720 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 37 M 2024-04-16T07:43:44.721 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 209 M 2024-04-16T07:43:44.721 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:43:44.977 INFO:teuthology.orchestra.run.smithi167.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 246 kB/s | 32 kB 00:00 2024-04-16T07:43:45.036 INFO:teuthology.orchestra.run.smithi167.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 489 kB/s | 17 kB 00:00 2024-04-16T07:43:45.069 INFO:teuthology.orchestra.run.smithi167.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-04-16T07:43:45.253 INFO:teuthology.orchestra.run.smithi167.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-04-16T07:43:45.420 INFO:teuthology.orchestra.run.smithi167.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-04-16T07:43:45.461 INFO:teuthology.orchestra.run.smithi167.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-04-16T07:43:45.587 INFO:teuthology.orchestra.run.smithi167.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.0 MB/s | 3.0 MB 00:00 2024-04-16T07:43:45.937 INFO:teuthology.orchestra.run.smithi167.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-04-16T07:43:46.129 INFO:teuthology.orchestra.run.smithi167.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-04-16T07:43:46.154 INFO:teuthology.orchestra.run.smithi167.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 906 kB/s | 195 kB 00:00 2024-04-16T07:43:47.113 INFO:teuthology.orchestra.run.smithi167.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 249 kB/s | 245 kB 00:00 2024-04-16T07:43:58.829 INFO:teuthology.orchestra.run.smithi167.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 542 kB/s | 7.4 MB 00:13 2024-04-16T07:43:58.833 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:58.833 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.6 MB/s | 37 MB 00:14 2024-04-16T07:43:58.878 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:43:58.904 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:43:58.904 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:43:59.262 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:43:59.262 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:43:59.972 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:00.051 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:44:00.091 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:44:00.194 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:44:00.237 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:44:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:44:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:44:00.521 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:44:01.176 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:44:01.300 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:44:02.537 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:44:03.567 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:44:03.603 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:44:06.146 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:44:06.146 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 1/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:44:06.147 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:44:06.564 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:44:06.564 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:06.565 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:06.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:44:07.545 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:29 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:07.697 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:07.698 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-04-16T07:44:07.698 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 49 k 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:44:07.699 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:44:07.700 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:44:07.700 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:44:07.700 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:44:07.700 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:07.700 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:44:07.700 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-04-16T07:44:07.700 INFO:teuthology.orchestra.run.smithi167.stdout:Install 13 Packages 2024-04-16T07:44:07.700 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:07.700 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 2.0 M 2024-04-16T07:44:07.701 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 26 M 2024-04-16T07:44:07.701 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:44:07.931 INFO:teuthology.orchestra.run.smithi167.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 177 kB/s | 24 kB 00:00 2024-04-16T07:44:07.956 INFO:teuthology.orchestra.run.smithi167.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2024-04-16T07:44:08.014 INFO:teuthology.orchestra.run.smithi167.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2024-04-16T07:44:08.040 INFO:teuthology.orchestra.run.smithi167.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-04-16T07:44:08.065 INFO:teuthology.orchestra.run.smithi167.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T07:44:08.090 INFO:teuthology.orchestra.run.smithi167.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-04-16T07:44:08.232 INFO:teuthology.orchestra.run.smithi167.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 106 kB/s | 15 kB 00:00 2024-04-16T07:44:08.258 INFO:teuthology.orchestra.run.smithi167.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 168 kB/s | 32 kB 00:00 2024-04-16T07:44:08.324 INFO:teuthology.orchestra.run.smithi167.stdout:(9/13): ceph-mgr-rook-19.0.0-3109.g30b0aa00.el9 93 kB/s | 49 kB 00:00 2024-04-16T07:44:08.491 INFO:teuthology.orchestra.run.smithi167.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 783 kB/s | 203 kB 00:00 2024-04-16T07:44:08.660 INFO:teuthology.orchestra.run.smithi167.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 262 kB/s | 59 kB 00:00 2024-04-16T07:44:08.660 INFO:teuthology.orchestra.run.smithi167.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-04-16T07:44:08.684 INFO:teuthology.orchestra.run.smithi167.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-04-16T07:44:08.686 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:08.686 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-04-16T07:44:08.687 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:44:08.700 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:44:08.700 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:44:08.793 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:44:08.793 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:44:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:44:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:44:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:44:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:44:09.687 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:44:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:44:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:44:09.955 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:44:10.684 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:44:10.735 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:44:10.769 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:44:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:44:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:44:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 1/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:44:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:44:11.960 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:44:11.961 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:44:11.961 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:44:11.961 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:44:11.961 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:44:11.961 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:44:11.961 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:11.961 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:12.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:44:12.787 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:34 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:12.942 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:12.942 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-04-16T07:44:12.942 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:44:12.942 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-04-16T07:44:12.942 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 142 k 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2024-04-16T07:44:12.943 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:12.944 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 6.8 M 2024-04-16T07:44:12.944 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 33 M 2024-04-16T07:44:12.944 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:44:13.754 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-04-16T07:44:14.254 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3109.g30b0aa00.e 192 kB/s | 142 kB 00:00 2024-04-16T07:44:14.288 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-04-16T07:44:14.372 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 613 kB/s | 51 kB 00:00 2024-04-16T07:44:14.472 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.0 MB/s | 5.8 MB 00:00 2024-04-16T07:44:14.664 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): python3-natsort-7.1.1-5.el9.noarch.rpm 141 kB/s | 58 kB 00:00 2024-04-16T07:44:14.667 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:14.667 INFO:teuthology.orchestra.run.smithi167.stdout:Total 3.9 MB/s | 6.8 MB 00:01 2024-04-16T07:44:14.680 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:44:14.691 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:44:14.691 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:44:14.780 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:44:14.781 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:44:15.092 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:44:15.315 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:44:15.850 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:44:15.927 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:44:15.990 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:44:16.028 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:44:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:44:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/6 2024-04-16T07:44:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:44:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:44:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:44:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:44:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:44:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:17.126 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:44:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:44:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:44:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:44:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:44:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:17.127 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:17.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-fuse 2024-04-16T07:44:17.989 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:39 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:18.144 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 871 k 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:18.145 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-04-16T07:44:18.146 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:18.146 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 950 k 2024-04-16T07:44:18.146 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 4.0 M 2024-04-16T07:44:18.146 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:44:18.309 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 485 kB/s | 79 kB 00:00 2024-04-16T07:44:20.425 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-fuse-19.0.0-3109.g30b0aa00.el9.x86_ 382 kB/s | 871 kB 00:02 2024-04-16T07:44:20.427 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:20.427 INFO:teuthology.orchestra.run.smithi167.stdout:Total 416 kB/s | 950 kB 00:02 2024-04-16T07:44:20.427 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:44:20.453 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:44:20.453 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:44:20.530 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:44:20.531 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:44:20.715 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:20.876 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:44:20.950 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:44:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:44:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/2 2024-04-16T07:44:21.964 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:44:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:21.965 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:44:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:21.965 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:22.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-volume 2024-04-16T07:44:22.882 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:44 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:23.037 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:23.038 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:23.038 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-16T07:44:23.038 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:23.038 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:44:23.038 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 273 k 2024-04-16T07:44:23.038 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-16T07:44:23.038 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T07:44:23.038 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:23.039 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:44:23.039 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:23.039 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-04-16T07:44:23.039 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:23.039 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 342 k 2024-04-16T07:44:23.039 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.5 M 2024-04-16T07:44:23.039 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:44:23.147 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 635 kB/s | 69 kB 00:00 2024-04-16T07:44:24.031 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-volume-19.0.0-3109.g30b0aa00.el9.no 275 kB/s | 273 kB 00:00 2024-04-16T07:44:24.032 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:24.033 INFO:teuthology.orchestra.run.smithi167.stdout:Total 343 kB/s | 342 kB 00:00 2024-04-16T07:44:24.033 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:44:24.043 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:44:24.043 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:44:24.090 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:44:24.090 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:44:24.306 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T07:44:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:44:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:44:24.550 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:44:24.550 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T07:44:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 1/2 2024-04-16T07:44:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T07:44:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:25.624 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:44:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T07:44:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:25.625 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:25.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados-devel 2024-04-16T07:44:26.511 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:48 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 125 k 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-16T07:44:26.668 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:26.669 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 125 k 2024-04-16T07:44:26.669 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 445 k 2024-04-16T07:44:26.669 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:44:27.347 INFO:teuthology.orchestra.run.smithi167.stdout:librados-devel-19.0.0-3109.g30b0aa00.el9.x86_64 184 kB/s | 125 kB 00:00 2024-04-16T07:44:27.348 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:27.348 INFO:teuthology.orchestra.run.smithi167.stdout:Total 184 kB/s | 125 kB 00:00 2024-04-16T07:44:27.348 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:44:27.354 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:44:27.354 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:44:27.412 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:44:27.412 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:44:27.565 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:27.671 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:28.136 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:28.432 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:44:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:28.432 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:28.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs2 2024-04-16T07:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:51 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:29.400 INFO:teuthology.orchestra.run.smithi167.stdout:Package libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:29.452 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:29.453 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-16T07:44:29.453 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:29.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs-devel 2024-04-16T07:44:30.098 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:52 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:30.252 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:30.252 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-16T07:44:30.253 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:30.253 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:44:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 32 k 2024-04-16T07:44:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:30.253 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:44:30.253 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:30.253 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-16T07:44:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:30.254 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 32 k 2024-04-16T07:44:30.254 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 142 k 2024-04-16T07:44:30.254 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:44:30.710 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs-devel-19.0.0-3109.g30b0aa00.el9.x86_6 69 kB/s | 32 kB 00:00 2024-04-16T07:44:30.711 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:30.711 INFO:teuthology.orchestra.run.smithi167.stdout:Total 69 kB/s | 32 kB 00:00 2024-04-16T07:44:30.711 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:44:30.715 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:44:30.715 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:44:30.745 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:44:30.746 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:44:30.839 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:31.351 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:31.612 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:31.612 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:31.612 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:31.612 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:44:31.613 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:31.613 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:31.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados2 2024-04-16T07:44:32.390 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:54 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:32.490 INFO:teuthology.orchestra.run.smithi167.stdout:Package librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:32.543 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:32.544 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-16T07:44:32.544 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:32.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librbd1 2024-04-16T07:44:33.186 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:55 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:33.285 INFO:teuthology.orchestra.run.smithi167.stdout:Package librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:33.337 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:33.338 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-16T07:44:33.339 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:33.405 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rados 2024-04-16T07:44:33.989 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:55 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:34.089 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:34.141 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:34.142 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-16T07:44:34.142 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:34.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rgw 2024-04-16T07:44:34.786 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:56 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:34.885 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:34.937 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:34.938 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-16T07:44:34.938 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:35.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-cephfs 2024-04-16T07:44:35.629 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:57 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:35.739 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:35.796 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:35.798 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-16T07:44:35.798 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:35.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rbd 2024-04-16T07:44:36.476 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:58 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:36.577 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:44:36.628 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:36.630 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-16T07:44:36.630 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:36.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-fuse 2024-04-16T07:44:37.273 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:59 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:37.425 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:37.426 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:37.426 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-04-16T07:44:37.426 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:37.426 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:44:37.426 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 90 k 2024-04-16T07:44:37.427 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:37.427 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:44:37.427 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:37.427 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-16T07:44:37.427 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:37.427 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 90 k 2024-04-16T07:44:37.427 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 313 k 2024-04-16T07:44:37.427 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:44:38.290 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-fuse-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 104 kB/s | 90 kB 00:00 2024-04-16T07:44:38.291 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:38.291 INFO:teuthology.orchestra.run.smithi167.stdout:Total 104 kB/s | 90 kB 00:00 2024-04-16T07:44:38.291 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:44:38.301 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:44:38.301 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:44:38.360 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:44:38.360 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:44:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:39.347 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:39.347 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:39.347 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:39.347 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:44:39.347 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:39.347 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:39.582 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-mirror 2024-04-16T07:44:40.177 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:07:02 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:40.336 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 3.4 M 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 29 M 2024-04-16T07:44:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:44:46.011 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-mirror-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 614 kB/s | 3.4 MB 00:05 2024-04-16T07:44:46.011 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:46.012 INFO:teuthology.orchestra.run.smithi167.stdout:Total 614 kB/s | 3.4 MB 00:05 2024-04-16T07:44:46.012 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:44:46.025 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:44:46.025 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:44:46.105 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:44:46.105 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:44:46.676 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:46.736 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:46.761 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:46.762 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:44:46.762 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:44:46.762 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:44:46.762 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:44:46.762 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:48.094 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:48.095 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:48.095 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:48.095 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:44:48.095 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:48.095 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:48.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-nbd 2024-04-16T07:44:48.936 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:07:10 ago on Tue 16 Apr 2024 07:37:38 AM UTC. 2024-04-16T07:44:49.091 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-16T07:44:49.091 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 175 k 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 175 k 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 673 k 2024-04-16T07:44:49.092 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-16T07:44:49.991 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-nbd-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 195 kB/s | 175 kB 00:00 2024-04-16T07:44:49.992 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:44:49.992 INFO:teuthology.orchestra.run.smithi167.stdout:Total 195 kB/s | 175 kB 00:00 2024-04-16T07:44:49.992 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-16T07:44:50.002 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-16T07:44:50.002 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-16T07:44:50.060 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-16T07:44:50.060 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-16T07:44:50.219 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-16T07:44:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:50.745 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:44:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:51.029 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-16T07:44:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:44:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:51.029 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-16T07:44:51.279 DEBUG:teuthology.parallel:result is None 2024-04-16T07:44:51.280 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:44:51.511 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:44:51.537 INFO:teuthology.orchestra.run.smithi079.stdout:19.0.0-3109.g30b0aa00.el9 2024-04-16T07:44:51.537 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3109.g30b0aa00.el9 2024-04-16T07:44:51.537 INFO:teuthology.task.install:The correct ceph version 19.0.0-3109.g30b0aa00 is installed. 2024-04-16T07:44:51.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:44:51.696 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:44:51.718 INFO:teuthology.orchestra.run.smithi167.stdout:19.0.0-3109.g30b0aa00.el9 2024-04-16T07:44:51.718 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3109.g30b0aa00.el9 2024-04-16T07:44:51.718 INFO:teuthology.task.install:The correct ceph version 19.0.0-3109.g30b0aa00 is installed. 2024-04-16T07:44:51.720 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T07:44:51.720 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:51.720 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:44:51.755 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:44:51.755 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:44:51.788 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T07:44:51.788 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:51.788 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:44:51.827 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:44:51.898 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:44:51.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:44:51.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:44:51.999 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T07:44:51.999 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:51.999 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:44:52.029 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:44:52.099 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:44:52.099 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:44:52.127 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:44:52.195 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T07:44:52.195 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:52.195 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:44:52.223 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:44:52.292 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:44:52.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:44:52.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:44:52.390 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T07:44:52.473 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T07:44:52.473 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /var/log/ceph 2024-04-16T07:44:52.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /var/log/ceph 2024-04-16T07:44:52.503 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T07:44:52.503 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T07:44:52.545 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T07:44:52.572 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T07:44:52.573 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T07:44:52.613 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T07:44:52.650 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-16T07:44:52.651 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T07:44:52.651 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962', 'branch': 'wip-matanb-crimson-replicated-april-15', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench}', 'email': None, 'first_in_suite': False, 'job_id': '7657962', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-replicated-april-15'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-replicated-april-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 7594, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-replicated-april-15', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'targets': {'smithi079.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA5UD4iWmcHqD0asquxePKd1zDgMEmVjKTp6tq2G1cvMf5S0DQQU9rBSKz2/86Sc2+mdSYsp1sjTz8YVmxtua0Y=', 'smithi167.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvF4CbbQpotyRwsdFpVtdhZM8X38sysgVEwOX2Waa1gG2kMjIKBCYqQIJe2ufBRAsWXlIguOU6QCW3lf1wy3uI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'full_sequential': [{'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}]}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_07:22:04', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T07:44:52.651 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T07:44:52.682 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T07:44:52.697 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T07:44:52.738 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T07:44:52.766 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:52.766 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:44:52.821 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:44:52.821 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-04-16T07:44:52.875 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:44:52.875 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:44:52.875 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-16T07:44:52.876 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:44:52.876 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:44:52.876 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-16 07:43:30.327287142 +0000 2024-04-16T07:44:52.876 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-16 07:36:38.385588101 +0000 2024-04-16T07:44:52.876 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-16 07:36:38.385588101 +0000 2024-04-16T07:44:52.876 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-16 07:36:38.385588101 +0000 2024-04-16T07:44:52.876 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:44:52.942 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-16T07:44:52.942 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-16T07:44:52.942 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000205507 s, 2.5 MB/s 2024-04-16T07:44:52.943 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:44:52.999 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-04-16T07:44:53.054 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:44:53.055 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:44:53.055 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-16T07:44:53.055 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:44:53.055 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:44:53.055 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-16 07:43:30.327287142 +0000 2024-04-16T07:44:53.055 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-16 07:36:38.730579769 +0000 2024-04-16T07:44:53.055 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-16 07:36:38.730579769 +0000 2024-04-16T07:44:53.055 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-16 07:36:38.730579769 +0000 2024-04-16T07:44:53.055 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:44:53.121 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-16T07:44:53.121 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-16T07:44:53.121 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000185917 s, 2.8 MB/s 2024-04-16T07:44:53.122 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:44:53.178 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-04-16T07:44:53.233 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:44:53.233 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:44:53.233 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-16T07:44:53.233 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:44:53.233 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:44:53.233 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-16 07:43:30.327287142 +0000 2024-04-16T07:44:53.233 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-16 07:36:39.063571727 +0000 2024-04-16T07:44:53.234 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-16 07:36:39.063571727 +0000 2024-04-16T07:44:53.234 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-16 07:36:39.063571727 +0000 2024-04-16T07:44:53.234 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:44:53.300 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-16T07:44:53.301 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-16T07:44:53.301 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000185203 s, 2.8 MB/s 2024-04-16T07:44:53.302 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:44:53.358 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-04-16T07:44:53.413 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:44:53.413 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:44:53.413 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-16T07:44:53.414 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:44:53.414 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:44:53.414 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-16 07:43:30.328287118 +0000 2024-04-16T07:44:53.414 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-16 07:36:39.421563081 +0000 2024-04-16T07:44:53.414 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-16 07:36:39.421563081 +0000 2024-04-16T07:44:53.414 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-16 07:36:39.421563081 +0000 2024-04-16T07:44:53.414 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:44:53.480 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-16T07:44:53.480 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-16T07:44:53.480 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000180262 s, 2.8 MB/s 2024-04-16T07:44:53.481 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:44:53.537 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T07:44:53.537 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:44:53.537 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:44:53.555 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:44:53.555 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2024-04-16T07:44:53.612 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:44:53.612 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:44:53.612 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-16T07:44:53.612 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:44:53.612 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:44:53.612 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-16 07:44:46.997994692 +0000 2024-04-16T07:44:53.613 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-16 07:36:46.693228719 +0000 2024-04-16T07:44:53.613 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-16 07:36:46.693228719 +0000 2024-04-16T07:44:53.613 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-04-16 07:36:46.693228719 +0000 2024-04-16T07:44:53.613 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:44:53.680 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-16T07:44:53.681 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-16T07:44:53.681 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000340157 s, 1.5 MB/s 2024-04-16T07:44:53.682 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:44:53.739 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2024-04-16T07:44:53.795 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:44:53.796 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:44:53.796 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-16T07:44:53.796 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:44:53.796 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:44:53.796 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-16 07:44:46.997994692 +0000 2024-04-16T07:44:53.796 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-16 07:36:47.063222596 +0000 2024-04-16T07:44:53.796 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-16 07:36:47.063222596 +0000 2024-04-16T07:44:53.796 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-04-16 07:36:47.063222596 +0000 2024-04-16T07:44:53.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:44:53.863 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-16T07:44:53.863 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-16T07:44:53.863 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000229486 s, 2.2 MB/s 2024-04-16T07:44:53.864 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:44:53.921 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2024-04-16T07:44:53.977 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:44:53.977 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:44:53.977 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-16T07:44:53.977 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:44:53.977 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:44:53.977 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-16 07:44:46.998994676 +0000 2024-04-16T07:44:53.977 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-16 07:36:47.422216663 +0000 2024-04-16T07:44:53.978 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-16 07:36:47.422216663 +0000 2024-04-16T07:44:53.978 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-04-16 07:36:47.422216663 +0000 2024-04-16T07:44:53.978 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:44:54.043 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-16T07:44:54.044 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-16T07:44:54.044 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000193527 s, 2.6 MB/s 2024-04-16T07:44:54.045 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:44:54.101 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2024-04-16T07:44:54.157 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:44:54.157 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:44:54.157 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-16T07:44:54.157 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:44:54.157 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:44:54.157 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-16 07:44:46.998994676 +0000 2024-04-16T07:44:54.157 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-16 07:36:47.822210051 +0000 2024-04-16T07:44:54.157 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-16 07:36:47.822210051 +0000 2024-04-16T07:44:54.157 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-04-16 07:36:47.822210051 +0000 2024-04-16T07:44:54.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:44:54.223 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-16T07:44:54.224 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-16T07:44:54.224 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000207861 s, 2.5 MB/s 2024-04-16T07:44:54.225 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:44:54.281 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T07:44:54.282 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-16T07:44:54.282 INFO:tasks.ceph:Generating config... 2024-04-16T07:44:54.283 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-16T07:44:54.283 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-16T07:44:54.283 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-16T07:44:54.283 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-16T07:44:54.283 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-16T07:44:54.283 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] log to file = False 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-16T07:44:54.284 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-16T07:44:54.285 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-16T07:44:54.286 INFO:tasks.ceph:[client.0] debug ms = 1 2024-04-16T07:44:54.286 INFO:tasks.ceph:[client.0] debug objecter = 20 2024-04-16T07:44:54.286 INFO:tasks.ceph:[client.0] debug rados = 20 2024-04-16T07:44:54.286 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:44:54.286 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-16T07:44:54.286 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-16T07:44:54.286 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-16T07:44:54.286 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-16T07:44:54.286 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-16T07:44:54.286 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-16T07:44:54.286 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-16T07:44:54.286 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-16T07:44:54.287 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-16T07:44:54.287 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-16T07:44:54.287 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-16T07:44:54.288 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-16T07:44:54.288 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-16T07:44:54.288 INFO:tasks.ceph:Setting up mon.a... 2024-04-16T07:44:54.288 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-16T07:44:54.354 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.keyring 2024-04-16T07:44:54.358 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-16T07:44:54.412 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:44:54.440 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.79')] 2024-04-16T07:44:54.440 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.79', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-16T07:44:54.441 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:54.441 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:44:54.496 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.79 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:44:54.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:44:54.587+0000 7fc02f22ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:54.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:44:54.587+0000 7fc02f22ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:54.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:44:54.587+0000 7fc02f22ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:44:54.587 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: generated fsid 8dbe1cb9-e93e-4590-9181-7b27048c6254 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = quincy 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 0 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:fsid 8dbe1cb9-e93e-4590-9181-7b27048c6254 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:last_changed 2024-04-16T07:44:54.588890+0000 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-04-16T07:44:54.588890+0000 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:min_mon_release 17 (quincy) 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:election_strategy: 1 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-16T07:44:54.588 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-16T07:44:54.589 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:44:54.605 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8dbe1cb9-e93e-4590-9181-7b27048c6254... 2024-04-16T07:44:54.606 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T07:44:54.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T07:44:54.698 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-04-16T07:44:54.698 INFO:teuthology.orchestra.run.smithi079.stdout: chdir = "" 2024-04-16T07:44:54.698 INFO:teuthology.orchestra.run.smithi079.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T07:44:54.698 INFO:teuthology.orchestra.run.smithi079.stdout: auth supported = cephx 2024-04-16T07:44:54.698 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.698 INFO:teuthology.orchestra.run.smithi079.stdout: filestore xattr use omap = true 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: mon clock drift allowed = 1.000 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: osd crush chooseleaf type = 0 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: auth debug = true 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on old message = true 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on bug = true 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: mon pg warn max object skew = 0 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T07:44:54.699 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default size = 2 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow primary affinity = true 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow pg remap = true 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on legacy crush tunables = false 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on crush straw calc version zero = false 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on no sortbitwise = false 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on osd down out interval zero = false 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on too few osds = false 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: mon_allow_pool_size_one = true 2024-04-16T07:44:54.700 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: osd default data pool replay window = 5 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: mon allow pool delete = true 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: mon cluster log file level = debug 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: debug asserts on shutdown = true 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: mon health detail to clog = false 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: mon host = 172.21.15.79 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: ms client mode = crc 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: ms cluster mode = crc 2024-04-16T07:44:54.701 INFO:teuthology.orchestra.run.smithi079.stdout: ms mon client mode = crc 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: ms mon cluster mode = crc 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: ms mon service mode = crc 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: ms service mode = crc 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_min_size = 2 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_size = 3 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 8dbe1cb9-e93e-4590-9181-7b27048c6254 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout:[osd] 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: osd journal size = 100 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub load threshold = 5.0 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub max interval = 120 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock profile = high_recovery_ops 2024-04-16T07:44:54.702 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: osd recover clone overlap = true 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: osd recovery max chunk = 1048576 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug shutdown = true 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug op order = true 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify stray on activate = true 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug trim objects = true 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: osd open classes on start = true 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug pg log writeout = true 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: osd deep scrub update digest min age = 30 2024-04-16T07:44:54.703 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: osd map max advance = 10 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: journal zero on create = true 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: filestore ondisk finisher threads = 3 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: filestore apply finisher threads = 3 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: bdev debug aio = true 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug misdirected ops = true 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: debug monc = 20 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore block size = 96636764160 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression algorithm = snappy 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression mode = aggressive 2024-04-16T07:44:54.704 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore fsck on mount = True 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore rocksdb cf = False 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: crimson osd obc lru size = 10 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluefs = 20 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluestore = 20 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: debug osd = 20 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: debug rocksdb = 10 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: err to stderr = True 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: log flush on exit = True 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: log to file = False 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: log to stderr = True 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T07:44:54.705 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd full ratio = 0.9 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd nearfull ratio = 0.8 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd blocked scrub grace period = 3600 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug reject backfill probability = 0.3 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd delete sleep = 1 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd failsafe full ratio = 0.95 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd max backfills = 3 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd max markdown count = 1000 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd objectstore = bluestore 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub min interval = 60 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd shutdown pgref assert = True 2024-04-16T07:44:54.706 INFO:teuthology.orchestra.run.smithi079.stdout: osd snap trim sleep = 2 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr] 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout: debug mgr = 20 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout: mgr/telemetry/nag = false 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.707 INFO:teuthology.orchestra.run.smithi079.stdout:[mon] 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: debug paxos = 20 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: mon data avail warn = 5 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: mon mgr mkfs grace = 240 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd reporter subtree level = osd 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd prime pg temp = true 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T07:44:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: # 1m isn't quite enough 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: mon_down_mkfs_grace = 2m 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_filestore_osds = false 2024-04-16T07:44:54.709 INFO:teuthology.orchestra.run.smithi079.stdout: mon min osdmap epochs = 50 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune interval = 2 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune min = 15 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune txsize = 2 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default crimson = True 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: paxos service trim min = 10 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout:[client] 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: rgw cache enabled = true 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable ops log = true 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable usage log = true 2024-04-16T07:44:54.710 INFO:teuthology.orchestra.run.smithi079.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.a] 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout: debug objecter = 20 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout: debug rados = 20 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2024-04-16T07:44:54.711 INFO:teuthology.orchestra.run.smithi079.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:44:54.712 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.3] 2024-04-16T07:44:54.712 INFO:teuthology.orchestra.run.smithi079.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:44:54.712 INFO:teuthology.orchestra.run.smithi167.stdout:[global] 2024-04-16T07:44:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: chdir = "" 2024-04-16T07:44:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T07:44:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: auth supported = cephx 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: filestore xattr use omap = true 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: mon clock drift allowed = 1.000 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: osd crush chooseleaf type = 0 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: auth debug = true 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on old message = true 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on bug = true 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T07:44:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: mon pg warn max object skew = 0 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default size = 2 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow primary affinity = true 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow pg remap = true 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on legacy crush tunables = false 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on crush straw calc version zero = false 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on no sortbitwise = false 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on osd down out interval zero = false 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on too few osds = false 2024-04-16T07:44:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon_allow_pool_size_one = true 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: osd default data pool replay window = 5 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon allow pool delete = true 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon cluster log file level = debug 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: debug asserts on shutdown = true 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon health detail to clog = false 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon host = 172.21.15.79 2024-04-16T07:44:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: ms client mode = crc 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: ms cluster mode = crc 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: ms mon client mode = crc 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: ms mon cluster mode = crc 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: ms mon service mode = crc 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: ms service mode = crc 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_min_size = 2 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_size = 3 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: fsid = 8dbe1cb9-e93e-4590-9181-7b27048c6254 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout:[osd] 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: osd journal size = 100 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub load threshold = 5.0 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub max interval = 120 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock profile = high_recovery_ops 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd recover clone overlap = true 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd recovery max chunk = 1048576 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug shutdown = true 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug op order = true 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify stray on activate = true 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug trim objects = true 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd open classes on start = true 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug pg log writeout = true 2024-04-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: osd deep scrub update digest min age = 30 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: osd map max advance = 10 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: journal zero on create = true 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: filestore ondisk finisher threads = 3 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: filestore apply finisher threads = 3 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: bdev debug aio = true 2024-04-16T07:44:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug misdirected ops = true 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug monc = 20 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore block size = 96636764160 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore compression algorithm = snappy 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore compression mode = aggressive 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore fsck on mount = True 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore rocksdb cf = False 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: crimson osd obc lru size = 10 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluefs = 20 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluestore = 20 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug osd = 20 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug rocksdb = 10 2024-04-16T07:44:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: err to stderr = True 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: log flush on exit = True 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: log to file = False 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: log to stderr = True 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd backfillfull_ratio = 0.85 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd full ratio = 0.9 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd nearfull ratio = 0.8 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: osd blocked scrub grace period = 3600 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug reject backfill probability = 0.3 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: osd delete sleep = 1 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: osd failsafe full ratio = 0.95 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: osd max backfills = 3 2024-04-16T07:44:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: osd max markdown count = 1000 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: osd objectstore = bluestore 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub min interval = 60 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: osd shutdown pgref assert = True 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: osd snap trim sleep = 2 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr] 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: debug mgr = 20 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:44:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: mgr/telemetry/nag = false 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout:[mon] 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: debug paxos = 20 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon data avail warn = 5 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon mgr mkfs grace = 240 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd reporter subtree level = osd 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd prime pg temp = true 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:44:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: # 1m isn't quite enough 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: mon_down_mkfs_grace = 2m 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_filestore_osds = false 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: mon min osdmap epochs = 50 2024-04-16T07:44:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: mon osdmap full prune interval = 2 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: mon osdmap full prune min = 15 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: mon osdmap full prune txsize = 2 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default crimson = True 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: paxos service trim min = 10 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout:[client] 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: rgw cache enabled = true 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable ops log = true 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable usage log = true 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.a] 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout:[client.0] 2024-04-16T07:44:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: debug objecter = 20 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: debug rados = 20 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.0] 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.1] 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.2] 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: crimson seastar cpu cores = 0-2 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.3] 2024-04-16T07:44:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: crimson seastar cpu cores = 3-5 2024-04-16T07:44:54.726 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-16T07:44:54.727 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-16T07:44:54.794 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T07:44:54.794 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:54.794 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-16T07:44:54.810 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:54.810 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-16T07:44:54.864 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi079.front.sepia.ceph.com 2024-04-16T07:44:54.865 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:54.865 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T07:44:54.865 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:44:54.945 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:44:54.945 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:44:55.001 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi167.front.sepia.ceph.com 2024-04-16T07:44:55.001 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:44:55.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T07:44:55.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:44:55.044 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:44:55.044 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:44:55.100 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T07:44:55.100 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T07:44:55.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:44:55.233 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:44:55.236 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T07:44:55.236 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T07:44:55.236 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T07:44:55.288 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-16T07:44:55.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-16T07:44:55.357 INFO:teuthology.orchestra.run.smithi167.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-16T07:44:55.373 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T07:44:55.374 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-16T07:44:55.374 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-16T07:44:55.402 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T07:44:55.402 INFO:tasks.ceph:role: osd.0 2024-04-16T07:44:55.402 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-16T07:44:55.402 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T07:44:55.477 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:44:55.478 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:44:55.478 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:44:55.478 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:44:55.478 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:44:55.478 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:44:55.478 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:44:55.478 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:44:55.478 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:44:55.478 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:44:55.991 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-04-16T07:44:55.993 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-04-16T07:44:55.993 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-16T07:44:56.142 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-16T07:44:56.177 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-16T07:44:56.247 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-16T07:44:56.247 INFO:tasks.ceph:role: osd.1 2024-04-16T07:44:56.247 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-16T07:44:56.247 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T07:44:56.323 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:44:56.323 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:44:56.323 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:44:56.324 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:44:56.324 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:44:56.324 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:44:56.324 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:44:56.324 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:44:56.324 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:44:56.324 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:44:56.771 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-04-16T07:44:56.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-04-16T07:44:56.773 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-16T07:44:56.867 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-16T07:44:56.900 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:44:57.209 INFO:teuthology.orchestra.run.smithi079.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:44:57.210 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-16 07:44:57,212 seastar - Reactor backend: linux-aio 2024-04-16T07:44:57.221 INFO:teuthology.orchestra.run.smithi079.stderr:==29039==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:44:57.345 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-16 07:44:57,346 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:44:57.345 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-16 07:44:57,346 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-16T07:44:57.345 INFO:teuthology.orchestra.run.smithi079.stderr:DEBUG 2024-04-16 07:44:57,347 [shard 0:main] seastar - reactor::drain 2024-04-16T07:44:57.888 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-16T07:44:57.889 INFO:teuthology.orchestra.run.smithi079.stderr:Suppressions used: 2024-04-16T07:44:57.889 INFO:teuthology.orchestra.run.smithi079.stderr: count bytes template 2024-04-16T07:44:57.889 INFO:teuthology.orchestra.run.smithi079.stderr: 1 8 MallocExtension::Register 2024-04-16T07:44:57.889 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-16T07:44:57.889 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-16T07:44:57.951 INFO:teuthology.orchestra.run.smithi079.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:44:57.955 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-16 07:44:57,956 seastar - Reactor backend: linux-aio 2024-04-16T07:44:57.964 INFO:teuthology.orchestra.run.smithi079.stderr:==29037==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:02.614 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-16T07:45:02.614 INFO:teuthology.orchestra.run.smithi079.stderr:Suppressions used: 2024-04-16T07:45:02.614 INFO:teuthology.orchestra.run.smithi079.stderr: count bytes template 2024-04-16T07:45:02.614 INFO:teuthology.orchestra.run.smithi079.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:02.614 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-16T07:45:02.614 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-16T07:45:02.614 INFO:teuthology.orchestra.run.smithi079.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 8dbe1cb9-e93e-4590-9181-7b27048c6254 2024-04-16T07:45:02.637 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T07:45:02.665 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:02.958 INFO:teuthology.orchestra.run.smithi079.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:02.959 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-16 07:45:02,961 seastar - Reactor backend: linux-aio 2024-04-16T07:45:02.964 INFO:teuthology.orchestra.run.smithi079.stderr:==29222==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:03.088 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-16 07:45:03,089 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:45:03.088 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-16 07:45:03,089 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-16T07:45:03.088 INFO:teuthology.orchestra.run.smithi079.stderr:DEBUG 2024-04-16 07:45:03,090 [shard 0:main] seastar - reactor::drain 2024-04-16T07:45:03.624 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-16T07:45:03.624 INFO:teuthology.orchestra.run.smithi079.stderr:Suppressions used: 2024-04-16T07:45:03.624 INFO:teuthology.orchestra.run.smithi079.stderr: count bytes template 2024-04-16T07:45:03.624 INFO:teuthology.orchestra.run.smithi079.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:03.624 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-16T07:45:03.624 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-16T07:45:03.683 INFO:teuthology.orchestra.run.smithi079.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:03.687 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-16 07:45:03,688 seastar - Reactor backend: linux-aio 2024-04-16T07:45:03.696 INFO:teuthology.orchestra.run.smithi079.stderr:==29220==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:08.352 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-16T07:45:08.352 INFO:teuthology.orchestra.run.smithi079.stderr:Suppressions used: 2024-04-16T07:45:08.352 INFO:teuthology.orchestra.run.smithi079.stderr: count bytes template 2024-04-16T07:45:08.352 INFO:teuthology.orchestra.run.smithi079.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:08.352 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-16T07:45:08.352 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-16T07:45:08.353 INFO:teuthology.orchestra.run.smithi079.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 8dbe1cb9-e93e-4590-9181-7b27048c6254 2024-04-16T07:45:08.375 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T07:45:08.403 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-16T07:45:08.436 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T07:45:08.436 INFO:tasks.ceph:role: osd.2 2024-04-16T07:45:08.436 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-16T07:45:08.436 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:45:08.515 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:45:09.025 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-16T07:45:09.028 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-16T07:45:09.028 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-16T07:45:09.160 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-16T07:45:09.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-16T07:45:09.272 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-16T07:45:09.272 INFO:tasks.ceph:role: osd.3 2024-04-16T07:45:09.272 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-16T07:45:09.272 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T07:45:09.348 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:45:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:45:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:45:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:45:09.349 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:45:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:45:09.349 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:45:09.349 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:45:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:45:09.349 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:45:09.794 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-16T07:45:09.796 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-16T07:45:09.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-04-16T07:45:09.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-16T07:45:09.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:10.274 INFO:teuthology.orchestra.run.smithi167.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:10.275 INFO:teuthology.orchestra.run.smithi167.stderr:INFO 2024-04-16 07:45:10,277 seastar - Reactor backend: linux-aio 2024-04-16T07:45:10.286 INFO:teuthology.orchestra.run.smithi167.stderr:==28722==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:10.411 INFO:teuthology.orchestra.run.smithi167.stderr:INFO 2024-04-16 07:45:10,413 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:45:10.411 INFO:teuthology.orchestra.run.smithi167.stderr:INFO 2024-04-16 07:45:10,413 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-16T07:45:10.412 INFO:teuthology.orchestra.run.smithi167.stderr:DEBUG 2024-04-16 07:45:10,414 [shard 0:main] seastar - reactor::drain 2024-04-16T07:45:10.960 INFO:teuthology.orchestra.run.smithi167.stderr:----------------------------------------------------- 2024-04-16T07:45:10.961 INFO:teuthology.orchestra.run.smithi167.stderr:Suppressions used: 2024-04-16T07:45:10.961 INFO:teuthology.orchestra.run.smithi167.stderr: count bytes template 2024-04-16T07:45:10.961 INFO:teuthology.orchestra.run.smithi167.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:10.961 INFO:teuthology.orchestra.run.smithi167.stderr:----------------------------------------------------- 2024-04-16T07:45:10.961 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-16T07:45:11.021 INFO:teuthology.orchestra.run.smithi167.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:11.025 INFO:teuthology.orchestra.run.smithi167.stderr:INFO 2024-04-16 07:45:11,026 seastar - Reactor backend: linux-aio 2024-04-16T07:45:11.033 INFO:teuthology.orchestra.run.smithi167.stderr:==28720==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:15.687 INFO:teuthology.orchestra.run.smithi167.stderr:----------------------------------------------------- 2024-04-16T07:45:15.687 INFO:teuthology.orchestra.run.smithi167.stderr:Suppressions used: 2024-04-16T07:45:15.687 INFO:teuthology.orchestra.run.smithi167.stderr: count bytes template 2024-04-16T07:45:15.687 INFO:teuthology.orchestra.run.smithi167.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:15.687 INFO:teuthology.orchestra.run.smithi167.stderr:----------------------------------------------------- 2024-04-16T07:45:15.688 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-16T07:45:15.688 INFO:teuthology.orchestra.run.smithi167.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 8dbe1cb9-e93e-4590-9181-7b27048c6254 2024-04-16T07:45:15.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T07:45:15.738 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:16.033 INFO:teuthology.orchestra.run.smithi167.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:16.034 INFO:teuthology.orchestra.run.smithi167.stderr:INFO 2024-04-16 07:45:16,035 seastar - Reactor backend: linux-aio 2024-04-16T07:45:16.039 INFO:teuthology.orchestra.run.smithi167.stderr:==28904==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:16.164 INFO:teuthology.orchestra.run.smithi167.stderr:INFO 2024-04-16 07:45:16,165 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:45:16.164 INFO:teuthology.orchestra.run.smithi167.stderr:INFO 2024-04-16 07:45:16,165 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-16T07:45:16.164 INFO:teuthology.orchestra.run.smithi167.stderr:DEBUG 2024-04-16 07:45:16,166 [shard 0:main] seastar - reactor::drain 2024-04-16T07:45:16.693 INFO:teuthology.orchestra.run.smithi167.stderr:----------------------------------------------------- 2024-04-16T07:45:16.694 INFO:teuthology.orchestra.run.smithi167.stderr:Suppressions used: 2024-04-16T07:45:16.694 INFO:teuthology.orchestra.run.smithi167.stderr: count bytes template 2024-04-16T07:45:16.694 INFO:teuthology.orchestra.run.smithi167.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:16.694 INFO:teuthology.orchestra.run.smithi167.stderr:----------------------------------------------------- 2024-04-16T07:45:16.694 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-16T07:45:16.749 INFO:teuthology.orchestra.run.smithi167.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:45:16.753 INFO:teuthology.orchestra.run.smithi167.stderr:INFO 2024-04-16 07:45:16,754 seastar - Reactor backend: linux-aio 2024-04-16T07:45:16.761 INFO:teuthology.orchestra.run.smithi167.stderr:==28902==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:45:21.422 INFO:teuthology.orchestra.run.smithi167.stderr:----------------------------------------------------- 2024-04-16T07:45:21.423 INFO:teuthology.orchestra.run.smithi167.stderr:Suppressions used: 2024-04-16T07:45:21.423 INFO:teuthology.orchestra.run.smithi167.stderr: count bytes template 2024-04-16T07:45:21.423 INFO:teuthology.orchestra.run.smithi167.stderr: 1 8 MallocExtension::Register 2024-04-16T07:45:21.423 INFO:teuthology.orchestra.run.smithi167.stderr:----------------------------------------------------- 2024-04-16T07:45:21.423 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-16T07:45:21.423 INFO:teuthology.orchestra.run.smithi167.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 8dbe1cb9-e93e-4590-9181-7b27048c6254 2024-04-16T07:45:21.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T07:45:21.473 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T07:45:21.473 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:45:21.473 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-16T07:45:21.506 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:45:21.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-16T07:45:21.577 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:45:21.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-16T07:45:21.606 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:45:21.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-16T07:45:21.674 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:45:21.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-16T07:45:21.742 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:45:21.742 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-16T07:45:21.761 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:45:21.761 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-16T07:45:21.797 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T07:45:21.797 DEBUG:teuthology.orchestra.run.smithi079:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T07:45:21.831 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB6LB5m3BD8BhAADtzNBfZiZIqa/qOtlDToOA== 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB/LB5mI1pOMhAAGk8oA/EzWe+miZMsA0oZuw== 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.x] 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB3LB5mFhgADhAATH6GdTPFEGDmYIG4ohqnEQ== 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQCHLB5m1qgXCxAArhomBOEN7KuoNZQXLM0iAQ== 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.3] 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQCMLB5mhBiCNhAAtdjwWhXwAfBrm+EDOWWYHQ== 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB3LB5mV0lJERAA3ZbHRnZHPPVT2faB3Xwrzg== 2024-04-16T07:45:21.832 INFO:teuthology.orchestra.run.smithi079.stdout:[client.1] 2024-04-16T07:45:21.833 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB3LB5mffFoFRAADQQ7NqlWrRa0VxglJkeK6w== 2024-04-16T07:45:21.833 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:45:21.943 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:45:22.008 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T07:45:22.071 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:45:22.134 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:45:22.198 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T07:45:22.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T07:45:22.330 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T07:45:22.330 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-16T07:45:22.359 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T07:45:22.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:22.474+0000 7fb201878a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:22.474+0000 7fb201878a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.490 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-16T07:45:22.518 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:45:22.573 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-16T07:45:22.574 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-16T07:45:22.574 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-16T07:45:22.615 INFO:tasks.ceph.mon.a:Started 2024-04-16T07:45:22.616 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-16T07:45:22.616 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-16T07:45:22.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-16T07:45:22.618 INFO:tasks.ceph.mgr.x:Started 2024-04-16T07:45:22.619 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T07:45:22.619 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd crush tunables default 2024-04-16T07:45:22.701 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:45:22.701+0000 7fa0d592ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.701 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:45:22.701+0000 7fa0d592ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.701 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:45:22.701+0000 7fa0d592ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.725 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:22.726+0000 7fc5bd634000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:22.725+0000 7f61be99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.735 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:22.735+0000 7fc5bd634000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.735 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:22.736+0000 7fc5bd634000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:22.736+0000 7f61be99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.806 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-04-16T07:45:22.818 INFO:tasks.ceph:check_enable_crimson: True 2024-04-16T07:45:22.818 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-16T07:45:22.819 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-16T07:45:22.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:22.923+0000 7fb8196ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:22.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:22.932+0000 7fb8196ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:23.011 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:45:23.012+0000 7fa0cd7f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-16T07:45:23.011 INFO:tasks.ceph.mon.a.smithi079.stderr:Please be aware that this feature is experimental, untested, 2024-04-16T07:45:23.011 INFO:tasks.ceph.mon.a.smithi079.stderr:unsupported, and may result in data corruption, data loss, 2024-04-16T07:45:23.011 INFO:tasks.ceph.mon.a.smithi079.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-16T07:45:23.011 INFO:tasks.ceph.mon.a.smithi079.stderr:feature with important data. 2024-04-16T07:45:23.012 INFO:tasks.ceph.mon.a.smithi079.stderr: 2024-04-16T07:45:23.200 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:23.200+0000 7fc5bd634000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T07:45:23.254 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:23.254+0000 7fc5bd634000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T07:45:23.790 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:23.791+0000 7fc5bd634000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T07:45:23.821 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-16T07:45:23.821 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:45:23.821 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-16T07:45:23.844 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:23.844+0000 7fc5bd634000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T07:45:23.852 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-16T07:45:23.852 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-16T07:45:23.888 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:23.889+0000 7fc5bd634000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T07:45:23.922 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:45:23.922 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-16T07:45:23.943 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:23.944+0000 7fc5bd634000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T07:45:23.951 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-16T07:45:23.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-16T07:45:23.987 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:23.988+0000 7fc5bd634000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T07:45:24.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new a3a8b979-b056-4896-89af-b4535cf4c672 0 2024-04-16T07:45:24.111 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:24.111+0000 7fc5bd634000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T07:45:24.167 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-16T07:45:24.162+0000 7f79c6c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:24.171 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-16T07:45:24.171+0000 7f79c6c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:24.186 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:24.187+0000 7fc5bd634000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T07:45:24.234 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:24.235+0000 7fc5bd634000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T07:45:24.251 INFO:teuthology.orchestra.run.smithi167.stdout:0 2024-04-16T07:45:24.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 56770dd9-36e1-4d48-84bd-214aeeffd060 1 2024-04-16T07:45:24.290 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:24.291+0000 7fc5bd634000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T07:45:24.341 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:24.341+0000 7fc5bd634000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T07:45:24.375 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-16T07:45:24.370+0000 7f871c8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:24.378 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-16T07:45:24.378+0000 7f871c8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:24.458 INFO:teuthology.orchestra.run.smithi167.stdout:1 2024-04-16T07:45:24.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new df0c762d-e963-4f0b-ad15-db4274a4d93b 2 2024-04-16T07:45:24.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-16T07:45:24.618+0000 7fb42f15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:24.626 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-16T07:45:24.627+0000 7fb42f15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:24.706 INFO:teuthology.orchestra.run.smithi167.stdout:2 2024-04-16T07:45:24.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 8f9b68d6-63a7-48b5-9fa7-a60682dbd93f 3 2024-04-16T07:45:24.742 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:24.743+0000 7fc5bd634000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T07:45:24.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-16T07:45:24.824+0000 7f57216ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:24.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-16T07:45:24.833+0000 7f57216ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:24.875 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:24.876+0000 7fc5bd634000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T07:45:24.912 INFO:teuthology.orchestra.run.smithi167.stdout:3 2024-04-16T07:45:24.923 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-16T07:45:24.963 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:45:24.963 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:45:24.989 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:24.990+0000 7fc5bd634000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T07:45:25.005 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:45:25.005 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-16T07:45:25.035 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:25.035+0000 7fc5bd634000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T07:45:25.041 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:45:25.041 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:45:25.084 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:45:25.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-16T07:45:25.117 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T07:45:25.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T07:45:25.151 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:25.152+0000 7fc5bd634000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:45:25.153 INFO:tasks.ceph.osd.2:Started 2024-04-16T07:45:25.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-04-16T07:45:25.186 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T07:45:25.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T07:45:25.229 INFO:tasks.ceph.osd.3:Started 2024-04-16T07:45:25.229 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:25.255 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:25.256+0000 7fc5bd634000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T07:45:25.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:25.344+0000 7f0b75ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:25.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:25.352+0000 7f0b75ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:45:25.421 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":7,"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","created":"2024-04-16T07:45:22.723906+0000","modified":"2024-04-16T07:45:24.912895+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:45:24.912895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a3a8b979-b056-4896-89af-b4535cf4c672","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"56770dd9-36e1-4d48-84bd-214aeeffd060","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df0c762d-e963-4f0b-ad15-db4274a4d93b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f9b68d6-63a7-48b5-9fa7-a60682dbd93f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:25.431 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:25.431+0000 7fc5bd634000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:45:25.432 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-16T07:45:25.432 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T07:45:25.478 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:25.478+0000 7fc5bd634000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T07:45:25.644 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:25.644+0000 7fc5bd634000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T07:45:25.835 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:25.940 INFO:teuthology.misc.health.smithi079.stderr:2024-04-16T07:45:25.935+0000 7f700d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:25.943 INFO:teuthology.misc.health.smithi079.stderr:2024-04-16T07:45:25.944+0000 7f700d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:26.012 INFO:teuthology.misc.health.smithi079.stdout: 2024-04-16T07:45:26.013 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":7,"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","created":"2024-04-16T07:45:22.723906+0000","modified":"2024-04-16T07:45:24.912895+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:45:24.912895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a3a8b979-b056-4896-89af-b4535cf4c672","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"56770dd9-36e1-4d48-84bd-214aeeffd060","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df0c762d-e963-4f0b-ad15-db4274a4d93b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f9b68d6-63a7-48b5-9fa7-a60682dbd93f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:26.023 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-16T07:45:26.544 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:26.544+0000 7fc5bd634000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T07:45:26.832 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:26.832+0000 7fc5bd634000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T07:45:26.889 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:26.890+0000 7fc5bd634000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T07:45:27.027 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:27.027+0000 7fc5bd634000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T07:45:28.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:45:28.037+0000 7fc55c302640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T07:45:32.426 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:32.535 INFO:teuthology.misc.health.smithi079.stderr:2024-04-16T07:45:32.532+0000 7f5e3df9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:32.539 INFO:teuthology.misc.health.smithi079.stderr:2024-04-16T07:45:32.539+0000 7f5e3df9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:32.727 INFO:teuthology.misc.health.smithi079.stdout: 2024-04-16T07:45:32.727 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","created":"2024-04-16T07:45:22.723906+0000","modified":"2024-04-16T07:45:32.048077+0000","last_up_change":"2024-04-16T07:45:30.037723+0000","last_in_change":"2024-04-16T07:45:24.912895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:45:31.054622+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3a8b979-b056-4896-89af-b4535cf4c672","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":2500588370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2500588370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":2500588370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2500588370}]},"public_addr":"172.21.15.79:6801/2500588370","cluster_addr":"172.21.15.79:6802/2500588370","heartbeat_back_addr":"172.21.15.79:6809/2500588370","heartbeat_front_addr":"172.21.15.79:6808/2500588370","state":["exists","up"]},{"osd":1,"uuid":"56770dd9-36e1-4d48-84bd-214aeeffd060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2556495089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":2556495089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":2556495089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2556495089}]},"public_addr":"172.21.15.79:6804/2556495089","cluster_addr":"172.21.15.79:6805/2556495089","heartbeat_back_addr":"172.21.15.79:6803/2556495089","heartbeat_front_addr":"172.21.15.79:6806/2556495089","state":["exists","up"]},{"osd":2,"uuid":"df0c762d-e963-4f0b-ad15-db4274a4d93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":4043769319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4043769319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":4043769319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6811","nonce":4043769319}]},"public_addr":"172.21.15.167:6803/4043769319","cluster_addr":"172.21.15.167:6804/4043769319","heartbeat_back_addr":"172.21.15.167:6812/4043769319","heartbeat_front_addr":"172.21.15.167:6811/4043769319","state":["exists","up"]},{"osd":3,"uuid":"8f9b68d6-63a7-48b5-9fa7-a60682dbd93f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6805","nonce":3999508535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":3999508535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6807","nonce":3999508535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":3999508535}]},"public_addr":"172.21.15.167:6805/3999508535","cluster_addr":"172.21.15.167:6806/3999508535","heartbeat_back_addr":"172.21.15.167:6807/3999508535","heartbeat_front_addr":"172.21.15.167:6802/3999508535","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:32.739 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-16T07:45:32.739 INFO:tasks.ceph:Creating RBD pool 2024-04-16T07:45:32.739 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-16T07:45:32.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:32.846+0000 7f3073ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:32.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:32.856+0000 7f3073ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'rbd' created 2024-04-16T07:45:33.070 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T07:45:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:33.198+0000 7ff58179d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:33.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:33.212+0000 7ff58179d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:34.057 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T07:45:34.080 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-16T07:45:34.081 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T07:45:34.081 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-16T07:45:34.081 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-16T07:45:34.081 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T07:45:34.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:34.239+0000 7f8e9766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:34.296+0000 7f8e9766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:34.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:45:34.554 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2607180345},{"type":"v1","addr":"172.21.15.167:6801","nonce":2607180345}]},"active_addr":"172.21.15.167:6801/2607180345","active_change":"2024-04-16T07:45:27.030460+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":887336062}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":1497168823}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":2341343143}]}]} 2024-04-16T07:45:34.558 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-16T07:45:34.558 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T07:45:34.558 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:34.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:34.672+0000 7fa92e11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:34.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:34.680+0000 7fa92e11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:34.873 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:45:34.873 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":12,"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","created":"2024-04-16T07:45:22.723906+0000","modified":"2024-04-16T07:45:34.055808+0000","last_up_change":"2024-04-16T07:45:30.037723+0000","last_in_change":"2024-04-16T07:45:24.912895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:45:31.054622+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:45:33.047075+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3a8b979-b056-4896-89af-b4535cf4c672","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":2500588370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2500588370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":2500588370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2500588370}]},"public_addr":"172.21.15.79:6801/2500588370","cluster_addr":"172.21.15.79:6802/2500588370","heartbeat_back_addr":"172.21.15.79:6809/2500588370","heartbeat_front_addr":"172.21.15.79:6808/2500588370","state":["exists","up"]},{"osd":1,"uuid":"56770dd9-36e1-4d48-84bd-214aeeffd060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2556495089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":2556495089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":2556495089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2556495089}]},"public_addr":"172.21.15.79:6804/2556495089","cluster_addr":"172.21.15.79:6805/2556495089","heartbeat_back_addr":"172.21.15.79:6803/2556495089","heartbeat_front_addr":"172.21.15.79:6806/2556495089","state":["exists","up"]},{"osd":2,"uuid":"df0c762d-e963-4f0b-ad15-db4274a4d93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":4043769319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4043769319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":4043769319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6811","nonce":4043769319}]},"public_addr":"172.21.15.167:6803/4043769319","cluster_addr":"172.21.15.167:6804/4043769319","heartbeat_back_addr":"172.21.15.167:6812/4043769319","heartbeat_front_addr":"172.21.15.167:6811/4043769319","state":["exists","up"]},{"osd":3,"uuid":"8f9b68d6-63a7-48b5-9fa7-a60682dbd93f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6805","nonce":3999508535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":3999508535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6807","nonce":3999508535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":3999508535}]},"public_addr":"172.21.15.167:6805/3999508535","cluster_addr":"172.21.15.167:6806/3999508535","heartbeat_back_addr":"172.21.15.167:6807/3999508535","heartbeat_front_addr":"172.21.15.167:6802/3999508535","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:34.886 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T07:45:34.886 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:45:34.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:34.994+0000 7f1473ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.001+0000 7f1473ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.191 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:45:35.191 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":13,"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","created":"2024-04-16T07:45:22.723906+0000","modified":"2024-04-16T07:45:35.058709+0000","last_up_change":"2024-04-16T07:45:30.037723+0000","last_in_change":"2024-04-16T07:45:24.912895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:45:31.054622+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:45:33.047075+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3a8b979-b056-4896-89af-b4535cf4c672","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":2500588370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2500588370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":2500588370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2500588370}]},"public_addr":"172.21.15.79:6801/2500588370","cluster_addr":"172.21.15.79:6802/2500588370","heartbeat_back_addr":"172.21.15.79:6809/2500588370","heartbeat_front_addr":"172.21.15.79:6808/2500588370","state":["exists","up"]},{"osd":1,"uuid":"56770dd9-36e1-4d48-84bd-214aeeffd060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2556495089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":2556495089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":2556495089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2556495089}]},"public_addr":"172.21.15.79:6804/2556495089","cluster_addr":"172.21.15.79:6805/2556495089","heartbeat_back_addr":"172.21.15.79:6803/2556495089","heartbeat_front_addr":"172.21.15.79:6806/2556495089","state":["exists","up"]},{"osd":2,"uuid":"df0c762d-e963-4f0b-ad15-db4274a4d93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":4043769319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4043769319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":4043769319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6811","nonce":4043769319}]},"public_addr":"172.21.15.167:6803/4043769319","cluster_addr":"172.21.15.167:6804/4043769319","heartbeat_back_addr":"172.21.15.167:6812/4043769319","heartbeat_front_addr":"172.21.15.167:6811/4043769319","state":["exists","up"]},{"osd":3,"uuid":"8f9b68d6-63a7-48b5-9fa7-a60682dbd93f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6805","nonce":3999508535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":3999508535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6807","nonce":3999508535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":3999508535}]},"public_addr":"172.21.15.167:6805/3999508535","cluster_addr":"172.21.15.167:6806/3999508535","heartbeat_back_addr":"172.21.15.167:6807/3999508535","heartbeat_front_addr":"172.21.15.167:6802/3999508535","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:45:35.204 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T07:45:35.204 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T07:45:35.205 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T07:45:35.205 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-16T07:45:35.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.323+0000 7fc52119d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.323+0000 7ffa08a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.323+0000 7fec4322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.323+0000 7fdc2186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.333+0000 7fdc2186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.333+0000 7fec4322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.334+0000 7fc52119d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.334+0000 7ffa08a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.366 INFO:teuthology.orchestra.run.smithi079.stdout:38654705669 2024-04-16T07:45:35.366 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-16T07:45:35.370 INFO:teuthology.orchestra.run.smithi079.stdout:38654705669 2024-04-16T07:45:35.370 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:45:35.374 INFO:teuthology.orchestra.run.smithi079.stdout:38654705669 2024-04-16T07:45:35.374 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:45:35.375 INFO:teuthology.orchestra.run.smithi079.stdout:38654705669 2024-04-16T07:45:35.375 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:45:35.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.479+0000 7ff0a2f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.489+0000 7ff0a2f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.529+0000 7f914002d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.529+0000 7f388186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.539+0000 7f388186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.539+0000 7f914002d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.546+0000 7f7ebb6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.565+0000 7f7ebb6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.736 INFO:teuthology.orchestra.run.smithi079.stdout:38654705669 2024-04-16T07:45:35.755 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.3 2024-04-16T07:45:35.755 INFO:teuthology.orchestra.run.smithi079.stdout:38654705669 2024-04-16T07:45:35.755 DEBUG:teuthology.parallel:result is None 2024-04-16T07:45:35.763 INFO:teuthology.orchestra.run.smithi079.stdout:38654705669 2024-04-16T07:45:35.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.0 2024-04-16T07:45:35.766 DEBUG:teuthology.parallel:result is None 2024-04-16T07:45:35.777 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.2 2024-04-16T07:45:35.777 DEBUG:teuthology.parallel:result is None 2024-04-16T07:45:35.811 INFO:teuthology.orchestra.run.smithi079.stdout:38654705669 2024-04-16T07:45:35.824 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.1 2024-04-16T07:45:35.824 DEBUG:teuthology.parallel:result is None 2024-04-16T07:45:35.824 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:45:35.824 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:35.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.973+0000 7f9f66e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:35.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:35.981+0000 7f9f66e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:36.167 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:45:36.168 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:45:36.180 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-16T07:45:35.069621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89898,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.161923+0000","last_change":"2024-04-16T07:45:34.163348+0000","last_active":"2024-04-16T07:45:34.161923+0000","last_peered":"2024-04-16T07:45:34.161923+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.158639+0000","last_became_peered":"2024-04-16T07:45:34.158639+0000","last_unstale":"2024-04-16T07:45:34.161923+0000","last_undegraded":"2024-04-16T07:45:34.161923+0000","last_fullsized":"2024-04-16T07:45:34.161923+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.146905+0000","last_change":"2024-04-16T07:45:34.148324+0000","last_active":"2024-04-16T07:45:34.146905+0000","last_peered":"2024-04-16T07:45:34.146905+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.142880+0000","last_became_peered":"2024-04-16T07:45:34.142880+0000","last_unstale":"2024-04-16T07:45:34.146905+0000","last_undegraded":"2024-04-16T07:45:34.146905+0000","last_fullsized":"2024-04-16T07:45:34.146905+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.164293+0000","last_change":"2024-04-16T07:45:34.166174+0000","last_active":"2024-04-16T07:45:34.164293+0000","last_peered":"2024-04-16T07:45:34.164293+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.156887+0000","last_became_peered":"2024-04-16T07:45:34.156887+0000","last_unstale":"2024-04-16T07:45:34.164293+0000","last_undegraded":"2024-04-16T07:45:34.164293+0000","last_fullsized":"2024-04-16T07:45:34.164293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.147585+0000","last_change":"2024-04-16T07:45:34.148487+0000","last_active":"2024-04-16T07:45:34.147585+0000","last_peered":"2024-04-16T07:45:34.147585+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.143925+0000","last_became_peered":"2024-04-16T07:45:34.143925+0000","last_unstale":"2024-04-16T07:45:34.147585+0000","last_undegraded":"2024-04-16T07:45:34.147585+0000","last_fullsized":"2024-04-16T07:45:34.147585+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.150378+0000","last_change":"2024-04-16T07:45:34.151791+0000","last_active":"2024-04-16T07:45:34.150378+0000","last_peered":"2024-04-16T07:45:34.150378+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.146414+0000","last_became_peered":"2024-04-16T07:45:34.146414+0000","last_unstale":"2024-04-16T07:45:34.150378+0000","last_undegraded":"2024-04-16T07:45:34.150378+0000","last_fullsized":"2024-04-16T07:45:34.150378+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.142779+0000","last_change":"2024-04-16T07:45:34.144570+0000","last_active":"2024-04-16T07:45:34.142779+0000","last_peered":"2024-04-16T07:45:34.142779+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.137013+0000","last_became_peered":"2024-04-16T07:45:34.137013+0000","last_unstale":"2024-04-16T07:45:34.142779+0000","last_undegraded":"2024-04-16T07:45:34.142779+0000","last_fullsized":"2024-04-16T07:45:34.142779+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.149920+0000","last_change":"2024-04-16T07:45:34.151074+0000","last_active":"2024-04-16T07:45:34.149920+0000","last_peered":"2024-04-16T07:45:34.149920+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.146399+0000","last_became_peered":"2024-04-16T07:45:34.146399+0000","last_unstale":"2024-04-16T07:45:34.149920+0000","last_undegraded":"2024-04-16T07:45:34.149920+0000","last_fullsized":"2024-04-16T07:45:34.149920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.168037+0000","last_change":"2024-04-16T07:45:34.169590+0000","last_active":"2024-04-16T07:45:34.168037+0000","last_peered":"2024-04-16T07:45:34.168037+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.163363+0000","last_became_peered":"2024-04-16T07:45:34.163363+0000","last_unstale":"2024-04-16T07:45:34.168037+0000","last_undegraded":"2024-04-16T07:45:34.168037+0000","last_fullsized":"2024-04-16T07:45:34.168037+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.714514+0000","last_change":"2024-04-16T07:45:33.114318+0000","last_active":"2024-04-16T07:45:34.714514+0000","last_peered":"2024-04-16T07:45:34.714514+0000","last_clean":"2024-04-16T07:45:34.714514+0000","last_became_active":"2024-04-16T07:45:33.107372+0000","last_became_peered":"2024-04-16T07:45:33.107372+0000","last_unstale":"2024-04-16T07:45:34.714514+0000","last_undegraded":"2024-04-16T07:45:34.714514+0000","last_fullsized":"2024-04-16T07:45:34.714514+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:36.181 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:36.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:36.289+0000 7f382cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:36.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:36.296+0000 7f382cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:45:36.483 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:45:36.496 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-16T07:45:35.069621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89898,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.161923+0000","last_change":"2024-04-16T07:45:34.163348+0000","last_active":"2024-04-16T07:45:34.161923+0000","last_peered":"2024-04-16T07:45:34.161923+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.158639+0000","last_became_peered":"2024-04-16T07:45:34.158639+0000","last_unstale":"2024-04-16T07:45:34.161923+0000","last_undegraded":"2024-04-16T07:45:34.161923+0000","last_fullsized":"2024-04-16T07:45:34.161923+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.146905+0000","last_change":"2024-04-16T07:45:34.148324+0000","last_active":"2024-04-16T07:45:34.146905+0000","last_peered":"2024-04-16T07:45:34.146905+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.142880+0000","last_became_peered":"2024-04-16T07:45:34.142880+0000","last_unstale":"2024-04-16T07:45:34.146905+0000","last_undegraded":"2024-04-16T07:45:34.146905+0000","last_fullsized":"2024-04-16T07:45:34.146905+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.164293+0000","last_change":"2024-04-16T07:45:34.166174+0000","last_active":"2024-04-16T07:45:34.164293+0000","last_peered":"2024-04-16T07:45:34.164293+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.156887+0000","last_became_peered":"2024-04-16T07:45:34.156887+0000","last_unstale":"2024-04-16T07:45:34.164293+0000","last_undegraded":"2024-04-16T07:45:34.164293+0000","last_fullsized":"2024-04-16T07:45:34.164293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.147585+0000","last_change":"2024-04-16T07:45:34.148487+0000","last_active":"2024-04-16T07:45:34.147585+0000","last_peered":"2024-04-16T07:45:34.147585+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.143925+0000","last_became_peered":"2024-04-16T07:45:34.143925+0000","last_unstale":"2024-04-16T07:45:34.147585+0000","last_undegraded":"2024-04-16T07:45:34.147585+0000","last_fullsized":"2024-04-16T07:45:34.147585+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.150378+0000","last_change":"2024-04-16T07:45:34.151791+0000","last_active":"2024-04-16T07:45:34.150378+0000","last_peered":"2024-04-16T07:45:34.150378+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.146414+0000","last_became_peered":"2024-04-16T07:45:34.146414+0000","last_unstale":"2024-04-16T07:45:34.150378+0000","last_undegraded":"2024-04-16T07:45:34.150378+0000","last_fullsized":"2024-04-16T07:45:34.150378+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.142779+0000","last_change":"2024-04-16T07:45:34.144570+0000","last_active":"2024-04-16T07:45:34.142779+0000","last_peered":"2024-04-16T07:45:34.142779+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.137013+0000","last_became_peered":"2024-04-16T07:45:34.137013+0000","last_unstale":"2024-04-16T07:45:34.142779+0000","last_undegraded":"2024-04-16T07:45:34.142779+0000","last_fullsized":"2024-04-16T07:45:34.142779+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.149920+0000","last_change":"2024-04-16T07:45:34.151074+0000","last_active":"2024-04-16T07:45:34.149920+0000","last_peered":"2024-04-16T07:45:34.149920+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.146399+0000","last_became_peered":"2024-04-16T07:45:34.146399+0000","last_unstale":"2024-04-16T07:45:34.149920+0000","last_undegraded":"2024-04-16T07:45:34.149920+0000","last_fullsized":"2024-04-16T07:45:34.149920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.168037+0000","last_change":"2024-04-16T07:45:34.169590+0000","last_active":"2024-04-16T07:45:34.168037+0000","last_peered":"2024-04-16T07:45:34.168037+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.163363+0000","last_became_peered":"2024-04-16T07:45:34.163363+0000","last_unstale":"2024-04-16T07:45:34.168037+0000","last_undegraded":"2024-04-16T07:45:34.168037+0000","last_fullsized":"2024-04-16T07:45:34.168037+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.714514+0000","last_change":"2024-04-16T07:45:33.114318+0000","last_active":"2024-04-16T07:45:34.714514+0000","last_peered":"2024-04-16T07:45:34.714514+0000","last_clean":"2024-04-16T07:45:34.714514+0000","last_became_active":"2024-04-16T07:45:33.107372+0000","last_became_peered":"2024-04-16T07:45:33.107372+0000","last_unstale":"2024-04-16T07:45:34.714514+0000","last_undegraded":"2024-04-16T07:45:34.714514+0000","last_fullsized":"2024-04-16T07:45:34.714514+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:36.496 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T07:45:36.497 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T07:45:36.497 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-16T07:45:36.497 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-16T07:45:36.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:36.609+0000 7f316046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:36.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:36.616+0000 7f316046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:45:36.840 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T07:45:36.852 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-16T07:45:36.853 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-16T07:45:36.864 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-16T07:45:36.864 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-matanb-crimson-replicated-april-15'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-replicated-april-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}} 2024-04-16T07:45:36.864 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-16T07:45:36.864 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-16T07:45:36.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:36.973+0000 7f2278c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:36.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:36.984+0000 7f2278c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.188 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-16T07:45:37.188 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:45:37.189 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:37.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:37.300+0000 7fd38f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:37.308+0000 7fd38f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.495 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:45:37.495 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:45:37.508 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-16T07:45:37.035236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89898,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.965615"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:35.076041+0000","last_change":"2024-04-16T07:45:34.163348+0000","last_active":"2024-04-16T07:45:35.076041+0000","last_peered":"2024-04-16T07:45:35.076041+0000","last_clean":"2024-04-16T07:45:35.076041+0000","last_became_active":"2024-04-16T07:45:34.158639+0000","last_became_peered":"2024-04-16T07:45:34.158639+0000","last_unstale":"2024-04-16T07:45:35.076041+0000","last_undegraded":"2024-04-16T07:45:35.076041+0000","last_fullsized":"2024-04-16T07:45:35.076041+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.146905+0000","last_change":"2024-04-16T07:45:34.148324+0000","last_active":"2024-04-16T07:45:34.146905+0000","last_peered":"2024-04-16T07:45:34.146905+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.142880+0000","last_became_peered":"2024-04-16T07:45:34.142880+0000","last_unstale":"2024-04-16T07:45:34.146905+0000","last_undegraded":"2024-04-16T07:45:34.146905+0000","last_fullsized":"2024-04-16T07:45:34.146905+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.164293+0000","last_change":"2024-04-16T07:45:34.166174+0000","last_active":"2024-04-16T07:45:34.164293+0000","last_peered":"2024-04-16T07:45:34.164293+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.156887+0000","last_became_peered":"2024-04-16T07:45:34.156887+0000","last_unstale":"2024-04-16T07:45:34.164293+0000","last_undegraded":"2024-04-16T07:45:34.164293+0000","last_fullsized":"2024-04-16T07:45:34.164293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.147585+0000","last_change":"2024-04-16T07:45:34.148487+0000","last_active":"2024-04-16T07:45:34.147585+0000","last_peered":"2024-04-16T07:45:34.147585+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.143925+0000","last_became_peered":"2024-04-16T07:45:34.143925+0000","last_unstale":"2024-04-16T07:45:34.147585+0000","last_undegraded":"2024-04-16T07:45:34.147585+0000","last_fullsized":"2024-04-16T07:45:34.147585+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.150378+0000","last_change":"2024-04-16T07:45:34.151791+0000","last_active":"2024-04-16T07:45:34.150378+0000","last_peered":"2024-04-16T07:45:34.150378+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.146414+0000","last_became_peered":"2024-04-16T07:45:34.146414+0000","last_unstale":"2024-04-16T07:45:34.150378+0000","last_undegraded":"2024-04-16T07:45:34.150378+0000","last_fullsized":"2024-04-16T07:45:34.150378+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:35.072696+0000","last_change":"2024-04-16T07:45:34.144570+0000","last_active":"2024-04-16T07:45:35.072696+0000","last_peered":"2024-04-16T07:45:35.072696+0000","last_clean":"2024-04-16T07:45:35.072696+0000","last_became_active":"2024-04-16T07:45:34.137013+0000","last_became_peered":"2024-04-16T07:45:34.137013+0000","last_unstale":"2024-04-16T07:45:35.072696+0000","last_undegraded":"2024-04-16T07:45:35.072696+0000","last_fullsized":"2024-04-16T07:45:35.072696+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.149920+0000","last_change":"2024-04-16T07:45:34.151074+0000","last_active":"2024-04-16T07:45:34.149920+0000","last_peered":"2024-04-16T07:45:34.149920+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.146399+0000","last_became_peered":"2024-04-16T07:45:34.146399+0000","last_unstale":"2024-04-16T07:45:34.149920+0000","last_undegraded":"2024-04-16T07:45:34.149920+0000","last_fullsized":"2024-04-16T07:45:34.149920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:35.077006+0000","last_change":"2024-04-16T07:45:34.169590+0000","last_active":"2024-04-16T07:45:35.077006+0000","last_peered":"2024-04-16T07:45:35.077006+0000","last_clean":"2024-04-16T07:45:35.077006+0000","last_became_active":"2024-04-16T07:45:34.163363+0000","last_became_peered":"2024-04-16T07:45:34.163363+0000","last_unstale":"2024-04-16T07:45:35.077006+0000","last_undegraded":"2024-04-16T07:45:35.077006+0000","last_fullsized":"2024-04-16T07:45:35.077006+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.714514+0000","last_change":"2024-04-16T07:45:33.114318+0000","last_active":"2024-04-16T07:45:34.714514+0000","last_peered":"2024-04-16T07:45:34.714514+0000","last_clean":"2024-04-16T07:45:34.714514+0000","last_became_active":"2024-04-16T07:45:33.107372+0000","last_became_peered":"2024-04-16T07:45:33.107372+0000","last_unstale":"2024-04-16T07:45:34.714514+0000","last_undegraded":"2024-04-16T07:45:34.714514+0000","last_fullsized":"2024-04-16T07:45:34.714514+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:37.508 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:45:37.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:37.617+0000 7fecdd95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:37.625+0000 7fecdd95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.810 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:45:37.810 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:45:37.823 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-16T07:45:37.035236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107912,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379448,"statfs":{"total":386547056640,"available":386436554752,"internally_reserved":0,"allocated":593920,"data_stored":1870024,"data_compressed":89898,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.965615"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:35.076041+0000","last_change":"2024-04-16T07:45:34.163348+0000","last_active":"2024-04-16T07:45:35.076041+0000","last_peered":"2024-04-16T07:45:35.076041+0000","last_clean":"2024-04-16T07:45:35.076041+0000","last_became_active":"2024-04-16T07:45:34.158639+0000","last_became_peered":"2024-04-16T07:45:34.158639+0000","last_unstale":"2024-04-16T07:45:35.076041+0000","last_undegraded":"2024-04-16T07:45:35.076041+0000","last_fullsized":"2024-04-16T07:45:35.076041+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.146905+0000","last_change":"2024-04-16T07:45:34.148324+0000","last_active":"2024-04-16T07:45:34.146905+0000","last_peered":"2024-04-16T07:45:34.146905+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.142880+0000","last_became_peered":"2024-04-16T07:45:34.142880+0000","last_unstale":"2024-04-16T07:45:34.146905+0000","last_undegraded":"2024-04-16T07:45:34.146905+0000","last_fullsized":"2024-04-16T07:45:34.146905+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.164293+0000","last_change":"2024-04-16T07:45:34.166174+0000","last_active":"2024-04-16T07:45:34.164293+0000","last_peered":"2024-04-16T07:45:34.164293+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.156887+0000","last_became_peered":"2024-04-16T07:45:34.156887+0000","last_unstale":"2024-04-16T07:45:34.164293+0000","last_undegraded":"2024-04-16T07:45:34.164293+0000","last_fullsized":"2024-04-16T07:45:34.164293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.147585+0000","last_change":"2024-04-16T07:45:34.148487+0000","last_active":"2024-04-16T07:45:34.147585+0000","last_peered":"2024-04-16T07:45:34.147585+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.143925+0000","last_became_peered":"2024-04-16T07:45:34.143925+0000","last_unstale":"2024-04-16T07:45:34.147585+0000","last_undegraded":"2024-04-16T07:45:34.147585+0000","last_fullsized":"2024-04-16T07:45:34.147585+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.150378+0000","last_change":"2024-04-16T07:45:34.151791+0000","last_active":"2024-04-16T07:45:34.150378+0000","last_peered":"2024-04-16T07:45:34.150378+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.146414+0000","last_became_peered":"2024-04-16T07:45:34.146414+0000","last_unstale":"2024-04-16T07:45:34.150378+0000","last_undegraded":"2024-04-16T07:45:34.150378+0000","last_fullsized":"2024-04-16T07:45:34.150378+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:35.072696+0000","last_change":"2024-04-16T07:45:34.144570+0000","last_active":"2024-04-16T07:45:35.072696+0000","last_peered":"2024-04-16T07:45:35.072696+0000","last_clean":"2024-04-16T07:45:35.072696+0000","last_became_active":"2024-04-16T07:45:34.137013+0000","last_became_peered":"2024-04-16T07:45:34.137013+0000","last_unstale":"2024-04-16T07:45:35.072696+0000","last_undegraded":"2024-04-16T07:45:35.072696+0000","last_fullsized":"2024-04-16T07:45:35.072696+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.149920+0000","last_change":"2024-04-16T07:45:34.151074+0000","last_active":"2024-04-16T07:45:34.149920+0000","last_peered":"2024-04-16T07:45:34.149920+0000","last_clean":"2024-04-16T07:45:33.051603+0000","last_became_active":"2024-04-16T07:45:34.146399+0000","last_became_peered":"2024-04-16T07:45:34.146399+0000","last_unstale":"2024-04-16T07:45:34.149920+0000","last_undegraded":"2024-04-16T07:45:34.149920+0000","last_fullsized":"2024-04-16T07:45:34.149920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:45:35.077006+0000","last_change":"2024-04-16T07:45:34.169590+0000","last_active":"2024-04-16T07:45:35.077006+0000","last_peered":"2024-04-16T07:45:35.077006+0000","last_clean":"2024-04-16T07:45:35.077006+0000","last_became_active":"2024-04-16T07:45:34.163363+0000","last_became_peered":"2024-04-16T07:45:34.163363+0000","last_unstale":"2024-04-16T07:45:35.077006+0000","last_undegraded":"2024-04-16T07:45:35.077006+0000","last_fullsized":"2024-04-16T07:45:35.077006+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:45:34.714514+0000","last_change":"2024-04-16T07:45:33.114318+0000","last_active":"2024-04-16T07:45:34.714514+0000","last_peered":"2024-04-16T07:45:34.714514+0000","last_clean":"2024-04-16T07:45:34.714514+0000","last_became_active":"2024-04-16T07:45:33.107372+0000","last_became_peered":"2024-04-16T07:45:33.107372+0000","last_unstale":"2024-04-16T07:45:34.714514+0000","last_undegraded":"2024-04-16T07:45:34.714514+0000","last_fullsized":"2024-04-16T07:45:34.714514+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:45:37.823 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T07:45:37.824 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-16T07:45:37.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:37.932+0000 7f6b83c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:37.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:37.940+0000 7f6b83c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.130 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 13 2024-04-16T07:45:38.130 INFO:teuthology.orchestra.run.smithi079.stdout:fsid 8dbe1cb9-e93e-4590-9181-7b27048c6254 2024-04-16T07:45:38.130 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-04-16T07:45:22.723906+0000 2024-04-16T07:45:38.130 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-04-16T07:45:35.058709+0000 2024-04-16T07:45:38.130 INFO:teuthology.orchestra.run.smithi079.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:crush_version 3 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:full_ratio 0.95 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:backfillfull_ratio 0.9 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:nearfull_ratio 0.85 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:require_min_compat_client luminous 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:min_compat_client jewel 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:require_osd_release squid 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:stretch_mode_enabled false 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:allow_crimson=true 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:max_osd 4 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.79:6801/2500588370 v2:172.21.15.79:6802/2500588370 exists,up a3a8b979-b056-4896-89af-b4535cf4c672 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.79:6804/2556495089 v2:172.21.15.79:6805/2556495089 exists,up 56770dd9-36e1-4d48-84bd-214aeeffd060 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.167:6803/4043769319 v2:172.21.15.167:6804/4043769319 exists,up df0c762d-e963-4f0b-ad15-db4274a4d93b 2024-04-16T07:45:38.131 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.167:6805/3999508535 v2:172.21.15.167:6806/3999508535 exists,up 8f9b68d6-63a7-48b5-9fa7-a60682dbd93f 2024-04-16T07:45:38.143 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.79:6801/2500588370 v2:172.21.15.79:6802/2500588370 exists,up a3a8b979-b056-4896-89af-b4535cf4c672', 'osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.79:6804/2556495089 v2:172.21.15.79:6805/2556495089 exists,up 56770dd9-36e1-4d48-84bd-214aeeffd060', 'osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.167:6803/4043769319 v2:172.21.15.167:6804/4043769319 exists,up df0c762d-e963-4f0b-ad15-db4274a4d93b', 'osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.167:6805/3999508535 v2:172.21.15.167:6806/3999508535 exists,up 8f9b68d6-63a7-48b5-9fa7-a60682dbd93f'] 2024-04-16T07:45:38.143 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-16T07:45:38.252 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-16T07:45:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mon.a", 2024-04-16T07:45:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster": "ceph", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "admin_socket_mode": "", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_client_required": "cephx, none", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_cluster_required": "cephx", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_debug": "true", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_service_required": "cephx", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_supported": "cephx", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_aio": "true", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_aio_poll_ms": "250", 2024-04-16T07:45:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_aio_reap_max": "16", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_async_discard_threads": "0", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_block_size": "4096", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_debug_aio": "false", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_debug_inflight_ios": "false", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_enable_discard": "false", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_flock_retry": "3", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_inject_crash": "0", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_ioring": "false", 2024-04-16T07:45:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_ioring_hipri": "false", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_type": "", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_alloc_size": "1048576", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_allocator": "hybrid", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_buffered_io": "true", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_check_for_zeros": "false", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_compact_log_sync": "false", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-16T07:45:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_max_log_runway": "4194304", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_max_prefetch": "1048576", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_min_flush_size": "524288", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_min_log_runway": "1048576", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_replay_recovery": "false", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_sync_write": "false", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_allocation_from_file": "true", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_allocator": "hybrid", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-16T07:45:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_create": "true", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_db_create": "false", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_db_path": "", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_db_size": "0", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_path": "", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_preallocate_file": "false", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_size": "107374182400", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_wal_create": "false", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_wal_path": "", 2024-04-16T07:45:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_wal_size": "100663296", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bluefs": "true", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_autotune": "true", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-16T07:45:38.258 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_size": "0", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_type": "2q", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_clone_cow": "true", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-16T07:45:38.259 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_mode": "none", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_csum_type": "crc32c", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_freelist": "false", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-16T07:45:38.260 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_small_allocations": "0", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_default_buffered_read": "true", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_default_buffered_write": "false", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-16T07:45:38.261 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fail_eio": "false", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_mount": "false", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_umount": "false", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-16T07:45:38.262 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_ignore_data_csum": "false", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_log_op_age": "5.000000", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_alloc_size": "0", 2024-04-16T07:45:38.263 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_blob_size": "0", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_deferred_txc": "32", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_min_alloc_size": "0", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_nid_prealloc": "1024", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_retry_disk_reads": "3", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_rocksdb_cf": "true", 2024-04-16T07:45:38.264 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_spdk_mem": "512", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-16T07:45:38.265 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_tracing": "false", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_zero_block_detection": "false", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-16T07:45:38.266 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_cluster_require_signatures": "false", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_cluster_require_version": "2", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_require_signatures": "false", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_require_version": "2", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_service_require_signatures": "false", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_service_require_version": "2", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_sign_messages": "true", 2024-04-16T07:45:38.267 INFO:teuthology.orchestra.run.smithi079.stdout: "chdir": "", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_acl_type": "", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_asio_thread_count": "2", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_cache_mid": "0.750000", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_cache_size": "16384", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_caps_release_delay": "5", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_check_pool_perm": "true", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_debug_force_sync_read": "false", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_debug_getattr_caps": "false", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_debug_inject_tick_delay": "0", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_die_on_failed_remount": "false", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_dirsize_rbytes": "true", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_force_lazyio": "false", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_fs": "", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-16T07:45:38.268 INFO:teuthology.orchestra.run.smithi079.stdout: "client_inject_release_failure": "false", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_max_inline_size": "4096", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mds_namespace": "", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_metadata": "", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mount_gid": "-1", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mount_timeout": "300", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mount_uid": "-1", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mountpoint": "/", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_notify_timeout": "10", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc": "true", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_max_dirty": "104857600", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_max_objects": "1000", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_size": "209715200", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_target_dirty": "8388608", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_permissions": "true", 2024-04-16T07:45:38.269 INFO:teuthology.orchestra.run.smithi079.stdout: "client_quota": "true", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_quota_df": "true", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_readahead_max_bytes": "0", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_readahead_max_periods": "4", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_readahead_min": "131072", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_reconnect_stale": "false", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_shutdown_timeout": "30", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_snapdir": ".snap", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_tick_interval": "1", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_trace": "", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_try_dentry_invalidate": "false", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_use_faked_inos": "false", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "client_use_random_mds": "false", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_graylog": "false", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_graylog_port": "12201", 2024-04-16T07:45:38.270 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_monitors": "default=true", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_syslog": "false", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_syslog_level": "info", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_addr": "-", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_network": "", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_network_interface": "", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "compressor_zlib_isal": "false", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "compressor_zlib_level": "5", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "compressor_zlib_winsize": "-15", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "compressor_zstd_level": "1", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_alien_op_num_threads": "6", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-16T07:45:38.271 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_osd_stat_interval": "0", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_seastar_cpu_cores": "", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location_hook": "", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location_hook_timeout": "10", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "daemonize": "false", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "daos_pool": "tank", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_alienstore": "0/5", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_allow_any_pool_priority": "false", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_asok": "1/5", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_asok_assert_abort": "false", 2024-04-16T07:45:38.272 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_asserts_on_shutdown": "true", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_auth": "20/20", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_bdev": "1/3", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_bluefs": "1/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_bluestore": "1/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_buffer": "0/1", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_ceph_exporter": "1/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_cephfs_mirror": "0/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_cephsqlite": "0/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_client": "0/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_compressor": "1/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_context": "0/1", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_crimson_interrupt": "0/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_crush": "1/1", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_crypto": "1/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_cyanstore": "0/5", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_deliberately_leak_memory": "false", 2024-04-16T07:45:38.273 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_disable_randomized_ping": "false", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_dpdk": "1/5", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_eventtrace": "1/5", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_filer": "0/1", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_filestore": "1/3", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_finisher": "1/1", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_fuse": "1/5", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_heartbeat_testing_span": "0", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_heartbeatmap": "1/5", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_javaclient": "1/5", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_journal": "1/3", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_journaler": "0/5", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_kstore": "1/5", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_lockdep": "0/1", 2024-04-16T07:45:38.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mclock": "1/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds": "1/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_balancer": "1/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_locker": "1/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_log": "1/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_log_expire": "1/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_migrator": "1/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_quiesce": "3/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_memstore": "1/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mgr": "2/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mgrc": "1/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mon": "20/20", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_monc": "0/10", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_ms": "1/1", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_none": "0/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_objclass": "0/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_objectcacher": "0/5", 2024-04-16T07:45:38.275 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_objecter": "0/1", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_optracker": "0/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_osd": "1/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_paxos": "20/20", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_perfcounter": "1/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_prioritycache": "1/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rados": "0/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rbd": "0/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rbd_mirror": "0/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rbd_pwl": "0/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rbd_replay": "0/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_refs": "0/0", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_reserver": "1/1", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw": "1/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_access": "1/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_datacache": "1/5", 2024-04-16T07:45:38.276 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_dbstore": "1/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_flight": "1/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_notification": "1/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_sync": "1/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rocksdb": "4/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_backref": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_cache": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_cleaner": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_device": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_epm": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_journal": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_lba": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_odata": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_omap": "0/5", 2024-04-16T07:45:38.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_onode": "0/5", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_t": "0/5", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_tm": "0/5", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_striper": "0/1", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_test": "0/5", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_throttle": "1/1", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_timer": "0/1", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_tp": "0/5", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_trace": "1/5", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "device_failure_prediction_mode": "none", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "err_to_graylog": "false", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "err_to_journald": "false", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "err_to_stderr": "true", 2024-04-16T07:45:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: "err_to_syslog": "false", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event_tracing": "false", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_addr": "0.0.0.0", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_http_port": "9926", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_prio_limit": "5", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_sort_metrics": "true", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_stats_period": "5", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "fake_statfs_for_testing": "0", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "fatal_signal_handlers": "true", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "filer_max_purge_ops": "10", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "filer_max_truncate_ops": "128", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_apply_finisher_threads": "1", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_blackhole": "false", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_btrfs_clone_range": "true", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_btrfs_snap": "true", 2024-04-16T07:45:38.279 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_caller_concurrency": "10", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_collect_device_partition_information": "true", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_commit_timeout": "600.000000", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_debug_inject_read_err": "false", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_debug_omap_check": "false", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_debug_verify_split": "false", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_dump_file": "", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fadvise": "true", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fail_eio": "true", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fd_cache_shards": "16", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fd_cache_size": "128", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fiemap": "false", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fiemap_threshold": "4096", 2024-04-16T07:45:38.280 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_inject_stall": "0", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_journal_parallel": "false", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_journal_trailing": "false", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_journal_writeahead": "false", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_kill_at": "0", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattrs": "0", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-16T07:45:38.281 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_xattr_value_size": "0", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_merge_threshold": "-10", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_odsync_write": "false", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_omap_backend": "rocksdb", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_omap_backend_path": "", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-16T07:45:38.282 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_op_thread_timeout": "60", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_op_threads": "2", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_punch_hole": "false", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_ops": "50", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_seek_data_hole": "false", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_sloppy_crc": "false", 2024-04-16T07:45:38.283 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_splice": "false", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_split_multiple": "2", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_split_rand_factor": "20", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_update_to": "1000", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_enable": "true", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-16T07:45:38.284 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_xfs_extsize": "false", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_zfs_snap": "false", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fio_dir": "/tmp/fio", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "8dbe1cb9-e93e-4590-9181-7b27048c6254", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_allow_other": "true", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_atomic_o_trunc": "true", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_big_writes": "true", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_debug": "false", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_default_permissions": "false", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_disable_pagecache": "false", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_max_write": "0", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_multithreaded": "true", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_require_active_mds": "true", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_set_user_groups": "true", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_splice_move": "true", 2024-04-16T07:45:38.285 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_splice_read": "true", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_splice_write": "true", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_use_invalidate_cb": "true", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "gss_target_name": "ceph", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "heartbeat_file": "", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "heartbeat_inject_failure": "0", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "heartbeat_interval": "5", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "host": "smithi079", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-16T07:45:38.286 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "inject_early_sigterm": "false", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "jaeger_agent_port": "6799", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "jaeger_tracing_enable": "false", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_aio": "true", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_align_min_size": "65536", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_block_align": "true", 2024-04-16T07:45:38.287 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_block_size": "4096", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_dio": "true", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_discard": "false", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_force_aio": "false", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_ignore_corruption": "false", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_max_write_bytes": "10485760", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_max_write_entries": "100", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_replay_from": "0", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_write_header_frequency": "0", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_zero_on_create": "false", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journaler_prefetch_periods": "10", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journaler_prezero_periods": "5", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "journaler_write_head_interval": "15", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "key": "", 2024-04-16T07:45:38.288 INFO:teuthology.orchestra.run.smithi079.stdout: "keyfile": "", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_backend": "rocksdb", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_default_stripe_size": "65536", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_fsck_on_mount": "false", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_max_bytes": "67108864", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_max_ops": "512", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_nid_prealloc": "1024", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_onode_map_size": "1024", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_sync_submit_transaction": "false", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_sync_transaction": "false", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "librados_thread_count": "2", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "lockdep": "false", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "lockdep_force_backtrace": "false", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "log_coarse_timestamps": "true", 2024-04-16T07:45:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_flush_on_exit": "false", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_graylog_host": "127.0.0.1", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_graylog_port": "12201", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_max_new": "1000", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_max_recent": "10000", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_stderr_prefix": "", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_stop_at_utilization": "0.970000", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": "true", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_graylog": "false", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_journald": "false", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_stderr": "false", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_syslog": "false", 2024-04-16T07:45:38.290 INFO:teuthology.orchestra.run.smithi079.stdout: "max_rotating_auth_attempts": "10", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_action_on_write_error": "1", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_alternate_name_max": "8192", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_asio_thread_count": "2", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_export_pin": "true", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_fragment_dirs": "true", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_fragment_interval": "5", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_interval": "10", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_max": "-1", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_max_until": "-1", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_merge_size": "50", 2024-04-16T07:45:38.291 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_midchunk": "0.300000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_min_start": "0.200000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_minchunk": "0.001000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_mode": "0", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_need_max": "1.200000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_need_min": "0.800000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_overload_epochs": "2", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_split_bits": "3", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_split_size": "10000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_target_decay": "10.000000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_beacon_grace": "15.000000", 2024-04-16T07:45:38.292 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_beacon_interval": "4.000000", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_mid": "0.700000", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_delay": "0", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_release_free_interval": "10", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_reservation": "0.050000", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_trim_interval": "1", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_trim_threshold": "262144", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-16T07:45:38.293 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_client_prealloc_inos": "1000", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_connect_bootstrapping": "false", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_damage_table_max_entries": "10000", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_debug_auth_pins": "false", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_debug_frag": "false", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_debug_scatterstat": "false", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_debug_subtrees": "false", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_decay_halflife": "5.000000", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_default_dir_hash": "2", 2024-04-16T07:45:38.294 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_defer_session_stale": "true", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_deny_all_reconnect": "false", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dir_keys_per_op": "16384", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dir_max_commit_size": "10", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dir_max_entries": "0", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dir_prefetch": "true", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dump_cache_on_map": "false", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_early_reply": "true", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_enable_op_tracker": "true", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_enforce_unique_name": "true", 2024-04-16T07:45:38.295 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_export_ephemeral_random": "true", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_health_summarize_threshold": "10", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_health_dummy": "false", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-16T07:45:38.296 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_migrator_session_race": "false", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_join_fs": "", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_journal_format": "1", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_create_at": "0", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_export_at": "0", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_import_at": "0", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_journal_at": "0", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_journal_expire_at": "0", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_journal_replay_at": "0", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_link_at": "0", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_mdstable_at": "0", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_openc_at": "0", 2024-04-16T07:45:38.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_rename_at": "0", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_shutdown_at": "0", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_event_large_threshold": "524288", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_events_per_segment": "1024", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_max_events": "-1", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_max_segments": "128", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_pause": "false", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_segment_size": "0", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_trim_threshold": "128", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-16T07:45:38.298 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_warn_factor": "2.000000", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_caps_per_client": "1048576", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_completed_flushes": "100000", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_completed_requests": "100000", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_export_size": "20971520", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_file_recover": "32", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_purge_files": "64", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_purge_ops": "8192", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_snaps_per_dir": "100", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_metrics_update_interval": "2", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_min_caps_per_client": "100", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_min_caps_working_set": "10000", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-16T07:45:38.299 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_numa_node": "-1", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_complaint_time": "30.000000", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_history_duration": "600", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_history_size": "20", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_history_slow_op_size": "20", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_log_threshold": "5", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_ping_grace": "15", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_ping_interval": "5", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_max_caps": "30000", 2024-04-16T07:45:38.300 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_warning_threshold": "262144", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_replay_interval": "1.000000", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_root_ino_gid": "0", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_root_ino_uid": "0", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-16T07:45:38.301 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_shutdown_check": "0", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_skip_ino": "0", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_snap_max_uid": "4294967294", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_snap_min_uid": "0", 2024-04-16T07:45:38.302 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_snap_rstat": "false", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_standby_replay_damaged": "false", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_symlink_recovery": "true", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_thrash_exports": "0", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_thrash_fragments": "0", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_tick_interval": "5.000000", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_valgrind_exit": "false", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_verify_backtrace": "1", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_verify_scatter": "false", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_wipe_sessions": "false", 2024-04-16T07:45:38.303 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool_debug": "false", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "memstore_device_bytes": "1073741824", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "memstore_page_set": "false", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "memstore_page_size": "65536", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_client_bytes": "134217728", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_client_messages": "512", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_disabled_modules": "", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-16T07:45:38.304 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_max_pg_creating": "1024", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_max_pg_num_change": "128", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_mds_bytes": "134217728", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_mds_messages": "128", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_mon_bytes": "134217728", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_mon_messages": "128", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_osd_bytes": "536870912", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_osd_messages": "8192", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_pool": "true", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_standby_modules": "true", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_stats_period": "5", 2024-04-16T07:45:38.305 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_stats_threshold": "5", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_tick_period": "2", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_allow_pool_delete": "true", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_allow_pool_size_one": "true", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_auth_validate_all_caps": "true", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_bytes": "104857600", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_directed_command_retry": "2", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_parallel": "3", 2024-04-16T07:45:38.306 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_log_interval": "1.000000", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_ping_interval": "10.000000", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_target_rank": "-1", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_file_level": "debug", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_file": "true", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-16T07:45:38.307 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_journald": "false", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_compact_on_bootstrap": "false", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_compact_on_start": "false", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_compact_on_trim": "true", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cpu_threads": "4", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_crush_min_required_version": "hammer", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_daemon_bytes": "419430400", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-16T07:45:38.308 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_data_avail_crit": "5", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_data_avail_warn": "5", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_data_size_warn": "16106127360", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_dump_json": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_dump_transactions": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_extra_checks": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_no_require_reef": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_no_require_squid": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-16T07:45:38.309 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_down_mkfs_grace": "120", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_down_uptime_grace": "60", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_election_default_strategy": "1", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_election_timeout": "5.000000", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_elector_ping_divisor": "2", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_enable_op_tracker": "true", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_fake_pool_delete": "false", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_force_quorum_join": "false", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_globalid_prealloc": "10000", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_detail_to_clog": "false", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_log_update_period": "5", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_max_detail": "50", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_to_clog": "true", 2024-04-16T07:45:38.310 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_to_clog_interval": "600", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_host": "172.21.15.79", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_host_override": "", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_initial_members": "", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_lease": "5.000000", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_log_full_interval": "50", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_log_max": "10000", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_log_max_summary": "50", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-16T07:45:38.311 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_log_epochs": "500", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_osd": "10000", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_pg_per_osd": "10000", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_pool_pg_num": "65536", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mds_force_trim_to": "0", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mds_skip_sanity": "false", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_memory_autotune": "true", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_memory_target": "2147483648", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_beacon_grace": "30", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_digest_period": "5", 2024-04-16T07:45:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_inactive_grace": "60", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_min_osdmap_epochs": "50", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_complaint_time": "30", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_history_duration": "600", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_history_size": "20", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_history_slow_op_size": "20", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_log_threshold": "5", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_auto_mark_in": "false", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-16T07:45:38.313 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_cache_size": "500", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_down_out_interval": "600", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_force_trim_to": "0", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-16T07:45:38.314 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_min_down_reporters": "2", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_report_timeout": "900", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-16T07:45:38.315 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_stuck_threshold": "60", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_probe_timeout": "2.000000", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_reweight_max_change": "0.050000", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_reweight_max_osds": "4", 2024-04-16T07:45:38.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_interval": "86400", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_max_keys": "100", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_timeout": "300", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_session_timeout": "300", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_smart_report_timeout": "5", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stat_smooth_intervals": "6", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stretch_pool_min_size": "2", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stretch_pool_size": "4", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-16T07:45:38.317 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_debug": "false", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_provider_kill_at": "0", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_requester_kill_at": "0", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_timeout": "60.000000", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_target_pg_per_osd": "100", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_tick_interval": "5", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_timecheck_interval": "300.000000", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_use_min_delay_socket": "false", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_older_version_delay": "604800", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-16T07:45:38.318 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_misplaced": "false", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_older_version": "true", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-16T07:45:38.319 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": "", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_tracing_enabled": "false", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_op_threads": "3", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_cm": "false", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_device_name": "", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_dscp": "96", 2024-04-16T07:45:38.320 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_local_gid": "", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_port_num": "1", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_sl": "3", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_support_srq": "true", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_type": "ib", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_reap_threshold": "5", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_before_connect": "false", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_ipv4": "true", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_ipv6": "false", 2024-04-16T07:45:38.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_msgr1": "true", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_msgr2": "true", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_port_max": "7568", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_port_min": "6800", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_retry_count": "3", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_retry_delay": "5", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_client": "false", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_mds": "false", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_mgr": "false", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_mon": "false", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_osd": "false", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_client_mode": "crc", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_cluster_mode": "crc", 2024-04-16T07:45:38.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_cluster_type": "", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_compress_secure": "false", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_connection_idle_timeout": "900", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_connection_ready_timeout": "10", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_crc_data": "true", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_crc_header": "true", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_bad_msg": "false", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_bug": "true", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_old_message": "true", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_skipped_message": "false", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_coremask": "0xF", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-16T07:45:38.323 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_enable_tso": "true", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_hugepages": "", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_lro": "true", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_memory_channel": "4", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_pmd": "", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_port_id": "0", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-16T07:45:38.324 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dump_on_send": "false", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_initial_backoff": "0.200000", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_delay_max": "1.000000", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_delay_type": "", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_network_congestion": "0", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_socket_failures": "0", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_learn_addr_from_peer": "true", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_max_accept_failures": "4", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_max_backoff": "15.000000", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_mon_client_mode": "crc", 2024-04-16T07:45:38.325 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_mon_cluster_mode": "crc", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_mon_service_mode": "crc", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_osd_compress_min_size": "1024", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_osd_compress_mode": "none", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_pq_min_cost": "65536", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_public_type": "", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_service_mode": "crc", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_tcp_listen_backlog": "512", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_tcp_nodelay": "true", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_tcp_rcvbuf": "0", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_type": "async+posix", 2024-04-16T07:45:38.326 INFO:teuthology.orchestra.run.smithi079.stdout: "no_config_file": "false", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_completion_locks_per_session": "32", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_inflight_ops": "1024", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_tick_interval": "5.000000", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_timeout": "10.000000", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objectstore_blackhole": "false", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "openssl_engine_opts": "", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_delay_time": "5.000000", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_hist_halflife": "1000", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_max_low_ops": "2", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_max_ops": "4", 2024-04-16T07:45:38.327 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_slop": "0.020000", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_asio_thread_count": "2", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_async_recovery_min_cost": "100", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backfill_scan_max": "512", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backfill_scan_min": "64", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backoff_on_degraded": "false", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backoff_on_peering": "false", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backoff_on_unfound": "true", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_beacon_report_interval": "300", 2024-04-16T07:45:38.328 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bench_duration": "30", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bench_max_block_size": "67108864", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_blkin_trace_all": "false", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_check_for_log_corruption": "false", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_class_update_on_start": "true", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_client_message_cap": "256", 2024-04-16T07:45:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_client_message_size_cap": "524288000", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_client_op_priority": "63", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_client_watch_timeout": "30", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_command_max_records": "256", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_compact_on_start": "false", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_crush_update_on_start": "true", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_crush_update_weight_set": "true", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-16T07:45:38.330 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_feed_pullee": "-1", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_misdirected_ops": "false", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_no_acting_change": "false", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_no_purge_strays": "false", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_op_order": "false", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-16T07:45:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_shutdown": "false", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_trim_objects": "false", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_verify_snaps": "false", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_keys": "1024", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_stride": "524288", 2024-04-16T07:45:38.332 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_default_notify_timeout": "30", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_delete_sleep": "0.000000", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_discard_disconnected_ops": "true", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_enable_op_tracker": "true", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_info": "true", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_shutdown": "true", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-16T07:45:38.333 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_function_tracing": "false", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_grace": "20", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_interval": "6", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_min_peers": "10", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_min_size": "2000", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_stale": "600", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-16T07:45:38.334 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_hit_set_max_size": "100000", 2024-04-16T07:45:38.338 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_hit_set_min_size": "1000", 2024-04-16T07:45:38.338 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_journal_size": "5120", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_kill_backfill_at": "0", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_cache_size": "50", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_dedup": "true", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_message_max": "40", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_share_max_epochs": "40", 2024-04-16T07:45:38.339 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_attr_name_len": "100", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_attr_size": "0", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_backfills": "1", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_markdown_count": "5", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_markdown_period": "600", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_object_name_len": "2048", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_object_namespace_len": "256", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_object_size": "134217728", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_pg_blocked_by": "16", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_pg_log_entries": "10000", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_pgls": "1024", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_push_cost": "8388608", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_push_objects": "10", 2024-04-16T07:45:38.340 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_scrubs": "3", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_trimming_pgs": "2", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_write_op_reply_len": "64", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_write_size": "90", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_profile": "balanced", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-16T07:45:38.341 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_base": "805306368", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_cache_min": "134217728", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-16T07:45:38.342 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_target": "4294967296", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_target_autotune": "false", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_min_pg_log_entries": "250", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_min_recovery_priority": "0", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_report_interval": "5", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_num_cache_shards": "32", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_num_op_tracker_shard": "32", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_numa_auto_affinity": "true", 2024-04-16T07:45:38.343 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_numa_node": "-1", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_numa_prefer_iface": "true", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_objecter_finishers": "1", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_objectstore": "bluestore", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_objectstore_fuse": "false", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_objectstore_tracing": "false", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_complaint_time": "30.000000", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_history_duration": "600", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_history_size": "20", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_history_slow_op_size": "20", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_log_threshold": "5", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_shards": "0", 2024-04-16T07:45:38.344 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_shards_hdd": "5", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_shards_ssd": "8", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_pq_min_cost": "65536", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_queue_cut_off": "high", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_thread_timeout": "15", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_open_classes_on_start": "true", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_os_flags": "0", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_peering_op_priority": "255", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_delete_cost": "1048576", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_delete_priority": "5", 2024-04-16T07:45:38.345 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_log_trim_max": "10000", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_log_trim_min": "100", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_crimson": "true", 2024-04-16T07:45:38.346 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flags": "0", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_min_size": "2", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_pg_num": "32", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_pgp_num": "0", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_read_ratio": "70", 2024-04-16T07:45:38.347 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_size": "3", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_type": "replicated", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_push_per_object_cost": "1000", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recover_clone_overlap": "true", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_cost": "20971520", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_active": "0", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-16T07:45:38.348 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_single_start": "1", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_op_priority": "3", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_priority": "5", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_sleep": "0.000000", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_repair_during_recovery": "false", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_requested_scrub_priority": "120", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_auto_repair": "false", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_begin_hour": "0", 2024-04-16T07:45:38.349 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_begin_week_day": "0", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_chunk_max": "25", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_chunk_min": "5", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_cost": "52428800", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_during_recovery": "false", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_end_hour": "0", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_end_week_day": "0", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_event_cost": "4096", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_invalid_stats": "true", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-16T07:45:38.350 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_max_preemptions": "5", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_priority": "5", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_sleep": "0.000000", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_skip_data_digest": "false", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_smart_report_timeout": "5", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_cost": "1048576", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_priority": "5", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-16T07:45:38.351 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_target_transaction_size": "30", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-16T07:45:38.352 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tracing": "false", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_use_stale_snap": "false", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "osdc_blkin_trace_all": "false", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_kill_at": "0", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_max_join_drift": "10", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_min": "500", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_min_wait": "0.050000", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_propose_interval": "1.000000", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_service_trim_max": "500", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_service_trim_min": "10", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_stash_full_interval": "25", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_trim_max": "500", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_trim_min": "250", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "perf": "true", 2024-04-16T07:45:38.353 INFO:teuthology.orchestra.run.smithi079.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "-", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrv": "", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "public_bind_addr": "-", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "public_network": "", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "public_network_interface": "", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "qat_compressor_enabled": "false", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "qat_compressor_session_max_number": "256", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "rados_mon_op_timeout": "0", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "rados_osd_op_timeout": "0", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "rados_tracing": "false", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_atime_update_interval": "60", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-16T07:45:38.354 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_balance_parent_reads": "false", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_balance_snap_reads": "false", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_blkin_trace_all": "false", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache": "true", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_policy": "writearound", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_size": "33554432", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_clone_copy_on_read": "false", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_compression_hint": "none", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_concurrent_management_ops": "10", 2024-04-16T07:45:38.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_clone_format": "auto", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_data_pool": "", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_features": "61", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_format": "2", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_map_options": "", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_order": "22", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_pool": "rbd", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_stripe_count": "0", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_stripe_unit": "0", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_enable_alloc_hint": "true", 2024-04-16T07:45:38.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_io_scheduler": "simple", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_order": "24", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_pool": "", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_splay_width": "4", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_localize_parent_reads": "false", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_localize_snap_reads": "false", 2024-04-16T07:45:38.357 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-16T07:45:38.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-16T07:45:38.359 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mtime_update_interval": "60", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_non_blocking_aio": "true", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_op_thread_timeout": "60", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_op_threads": "1", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_parent_cache_enabled": "false", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_plugins": "", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_bps_burst": "0", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_bps_limit": "0", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_exclude_ops": "0", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_iops_burst": "0", 2024-04-16T07:45:38.360 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_iops_limit": "0", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-16T07:45:38.361 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_read_from_replica_policy": "default", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_skip_partial_discard": "true", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_tracing": "false", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_validate_names": "true", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_validate_pool": "true", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "restapi_base_url": "", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "restapi_log_level": "", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_acl_grants_max_num": "100", 2024-04-16T07:45:38.362 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_admin_entry": "admin", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_backend_store": "rados", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_barbican_url": "", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_beast_enable_async": "true", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_counters_cache": "false", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-16T07:45:38.363 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cache_enabled": "true", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cache_expiry_interval": "900", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cache_lru_size": "10000", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_config_store": "rados", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_content_length_compat": "false", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_copy_obj_progress": "true", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cors_rules_max_num": "100", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cross_domain_policy": "", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_addr": "", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-16T07:45:38.364 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_password": "", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_username": "", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_require_ssl": "true", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-16T07:45:38.365 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_addr": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_auth": "token", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_namespace": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_prefix": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-16T07:45:38.366 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_token_file": "", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_buffersize": "524288", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_low_speed_time": "300", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-16T07:45:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_port": "6379", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_log_changes_size": "1000", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_log_num_shards": "128", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_log_window": "30", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-16T07:45:38.368 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-16T07:45:38.369 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dns_name": "", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dns_s3website_name": "", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dynamic_resharding": "true", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_gc_threads": "true", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_lc_threads": "true", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_ops_log": "false", 2024-04-16T07:45:38.370 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_quota_threads": "true", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_static_website": "false", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_usage_log": "false", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enforce_swift_acls": "true", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_exit_timeout_secs": "120", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_expose_bucket": "false", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_extended_http_attrs": "", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_filter": "none", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_frontends": "beast port=7480", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_deferred": "50", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_objs": "32", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-16T07:45:38.371 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_processor_period": "3600", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_init_timeout": "300", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_kafka_connection_idle": "300", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-16T07:45:38.372 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_domain": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_password": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_password_path": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_project": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_tenant": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_token": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_token_path": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_user": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_api_version": "2", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_domain": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_password": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_project": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_user": "", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-16T07:45:38.373 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_url": "", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_debug_interval": "-1", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_lock_max_time": "90", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_max_objs": "32", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_max_rules": "1000", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_max_worker": "3", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_thread_delay": "0", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_dnattr": "uid", 2024-04-16T07:45:38.374 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_searchfilter": "", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_uri": "", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_log_http_headers": "", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_log_object_name_utc": "false", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-16T07:45:38.375 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_attr_name_len": "0", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_attr_size": "0", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_chunk_size": "4194304", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_listing_results": "1000", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_notify_retries": "10", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_put_param_size": "1048576", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_put_size": "5368709120", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_slo_entries": "1000", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_md_log_max_shards": "64", 2024-04-16T07:45:38.376 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_mp_lock_max_time": "600", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_max_gc": "300", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-16T07:45:38.377 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_num_async_rados_threads": "32", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_num_control_oids": "8", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_numa_node": "-1", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_objexp_chunk_size": "100", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_objexp_gc_interval": "600", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_op_thread_timeout": "600", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_op_tracing": "false", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_opa_token": "", 2024-04-16T07:45:38.378 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_opa_url": "", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_opa_verify_ssl": "true", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ops_log_rados": "false", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ops_log_socket_path": "", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_period_push_interval": "2.000000", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_cache_lanes": "3", 2024-04-16T07:45:38.379 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_cache_partitions": "3", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_print_continue": "true", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_rados_tracing": "false", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_realm": "", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_realm_id": "", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-16T07:45:38.380 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_region": "", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_request_uri": "", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_batch_size": "64", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_max_aio": "128", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_num_logs": "16", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_thread_interval": "600", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_resolve_cname": "false", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-16T07:45:38.381 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_run_sync_thread": "true", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_scheduler_type": "throttler", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_script_uri": "", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_service_provider_name": "", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_client_id": "", 2024-04-16T07:45:38.382 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_client_secret": "", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_entry": "sts", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_key": "sts", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_min_session_duration": "900", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_token_introspection_url": "", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_account_in_url": "false", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_auth_entry": "auth", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_auth_url": "", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_custom_header": "", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_need_stats": "true", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_tenant_name": "", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_token_expiration": "86400", 2024-04-16T07:45:38.383 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_url": "", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_url_prefix": "swift", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_lease_period": "120", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-16T07:45:38.384 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_thread_pool_size": "512", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_comment": "", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_createby": "", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_encoding": "", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_flag": "false", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_origin": "", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_tracker": "", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_trust_forwarded_https": "false", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_usage_max_shards": "32", 2024-04-16T07:45:38.385 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_usage_max_user_shards": "1", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_use_opa_authz": "false", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_counters_cache": "false", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_max_buckets": "1000", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_policies_max_num": "100", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_unique_email": "true", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_verify_ssl": "true", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zone": "", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zone_id": "", 2024-04-16T07:45:38.386 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zonegroup": "", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zonegroup_id": "", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rgwlc_auto_session_clear": "true", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_block_size": "4096", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_size": "536870912", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-16T07:45:38.387 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_index_type": "binary_search", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_partition_filters": "false", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_perf": "false", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "run_dir": "/var/run/ceph", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_block_create": "true", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_cache_lru_size": "67108864", 2024-04-16T07:45:38.388 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_default_max_object_size": "16777216", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_device_size": "53687091200", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_journal_batch_capacity": "16", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_main_device_type": "SSD", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_segment_size": "67108864", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "set_keepcaps": "false", 2024-04-16T07:45:38.389 INFO:teuthology.orchestra.run.smithi079.stdout: "setgroup": "", 2024-04-16T07:45:38.390 INFO:teuthology.orchestra.run.smithi079.stdout: "setuser": "", 2024-04-16T07:45:38.390 INFO:teuthology.orchestra.run.smithi079.stdout: "setuser_match_path": "", 2024-04-16T07:45:38.390 INFO:teuthology.orchestra.run.smithi079.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-16T07:45:38.390 INFO:teuthology.orchestra.run.smithi079.stdout: "thp": "false", 2024-04-16T07:45:38.390 INFO:teuthology.orchestra.run.smithi079.stdout: "threadpool_default_timeout": "60", 2024-04-16T07:45:38.390 INFO:teuthology.orchestra.run.smithi079.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-16T07:45:38.390 INFO:teuthology.orchestra.run.smithi079.stdout: "throttler_perf_counter": "true" 2024-04-16T07:45:38.390 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-16T07:45:38.392 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-16T07:45:38.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:38.502+0000 7f830795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:38.507+0000 7f830795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.521 INFO:teuthology.orchestra.run.smithi079.stdout:{} 2024-04-16T07:45:38.521 INFO:teuthology.orchestra.run.smithi079.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-16T07:45:38.532 DEBUG:teuthology.orchestra.run.smithi079:> type ceph-objectstore-tool 2024-04-16T07:45:38.546 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-16T07:45:38.547 DEBUG:teuthology.orchestra.run.smithi167:> type ceph-objectstore-tool 2024-04-16T07:45:38.565 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-16T07:45:38.565 INFO:teuthology.run_tasks:Running task full_sequential... 2024-04-16T07:45:38.575 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-16T07:45:38.576 INFO:tasks.radosbench:Beginning radosbench... 2024-04-16T07:45:38.576 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-16T07:45:38.576 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-16T07:45:38.577 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-16T07:45:38.577 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-16T07:45:38.577 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:38.577 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3] 2024-04-16T07:45:38.577 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T07:45:38.577 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:45:38.577 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-16T07:45:38.578 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:38.647 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T07:45:38.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:38.699+0000 7f384d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:38.704+0000 7f384d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:38.699+0000 7f4361664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:38.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:38.710+0000 7f4361664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'unique_pool_0' created 2024-04-16T07:45:39.197 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-16T07:45:39.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:39.311+0000 7f16ad82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:39.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:39.324+0000 7f16ad82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:40.185 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-16T07:45:41.207 DEBUG:teuthology.orchestra.run.smithi079:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-04-16T07:45:41.210 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-16T07:45:41.210 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-16T07:45:41.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:hints = 1 2024-04-16T07:45:41.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-16T07:45:41.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_31310 2024-04-16T07:45:41.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:45:41.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-04-16T07:45:42.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1 16 26 10 0.624935 0.625 0.0190768 0.0340228 2024-04-16T07:45:43.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2 16 26 10 0.312456 0 - 0.0340228 2024-04-16T07:45:43.648 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3] 2024-04-16T07:45:43.648 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:43.648 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-16T07:45:43.648 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:45:43.648 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:45:43.652 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:45:43.652 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:45:43.767 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:45:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:43.771 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-16T07:45:44.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3 16 26 10 0.208305 0 - 0.0340228 2024-04-16T07:45:45.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 4 16 26 10 0.156229 0 - 0.0340228 2024-04-16T07:45:46.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 5 16 26 10 0.124984 0 - 0.0340228 2024-04-16T07:45:47.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 6 16 26 10 0.104154 0 - 0.0340228 2024-04-16T07:45:48.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 7 16 26 10 0.0892753 0 - 0.0340228 2024-04-16T07:45:48.772 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:45:48.880 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-16T07:45:48.880 INFO:teuthology.orchestra.run.smithi079.stdout: "ops_in_flight": [] 2024-04-16T07:45:48.880 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-16T07:45:49.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 8 16 26 10 0.0781161 0 - 0.0340228 2024-04-16T07:45:50.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 9 16 66 50 0.347183 0.3125 0.0147417 1.84416 2024-04-16T07:45:50.804 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:50.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:50.917+0000 7fa7d2e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:50.922+0000 7fa7d2e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:51.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 10 16 66 50 0.312464 0 - 1.84416 2024-04-16T07:45:52.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 11 16 66 50 0.284059 0 - 1.84416 2024-04-16T07:45:53.021 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:45:53.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:53.142+0000 7f5f31dfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:53.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:53.147+0000 7f5f31dfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:53.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 12 16 66 50 0.260387 0 - 1.84416 2024-04-16T07:45:53.890 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 0] 2024-04-16T07:45:53.890 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:53.890 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [1, 2, 3, 0] 2024-04-16T07:45:53.890 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T07:45:53.891 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:45:53.945 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T07:45:54.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 13 16 66 50 0.240357 0 - 1.84416 2024-04-16T07:45:55.249 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:45:55.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 14 16 66 50 0.223189 0 - 1.84416 2024-04-16T07:45:55.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:55.371+0000 7ff2638ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:55.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:45:55.377+0000 7ff2638ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:45:56.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 15 16 66 50 0.208309 0 - 1.84416 2024-04-16T07:45:57.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 16 16 66 50 0.195289 0 - 1.84416 2024-04-16T07:45:58.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 17 16 66 50 0.183802 0 - 1.84416 2024-04-16T07:45:58.947 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [2, 3, 0] 2024-04-16T07:45:58.947 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:45:58.947 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-16T07:45:58.947 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:45:58.947 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:45:58.951 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:45:58.951 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:45:59.066 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:45:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:45:59.069 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:45:59.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 18 16 66 50 0.173591 0 - 1.84416 2024-04-16T07:46:00.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 19 16 66 50 0.164455 0 - 1.84416 2024-04-16T07:46:01.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:46:01.282161+0000 min lat: 0.0147417 max lat: 8.30948 avg lat: 1.84416 2024-04-16T07:46:01.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:46:01.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 20 16 66 50 0.156232 0 - 1.84416 2024-04-16T07:46:02.235 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-16T07:46:02.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 21 16 66 50 0.148792 0 - 1.84416 2024-04-16T07:46:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:02.426 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:03.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 22 16 66 50 0.142028 0 - 1.84416 2024-04-16T07:46:04.071 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:46:04.180 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-16T07:46:04.180 INFO:teuthology.orchestra.run.smithi079.stdout: "ops_in_flight": [] 2024-04-16T07:46:04.181 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-16T07:46:04.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 23 16 66 50 0.135853 0 - 1.84416 2024-04-16T07:46:04.427 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:04.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:04.556+0000 7f928da2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:04.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:04.563+0000 7f928da2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:05.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 24 16 135 119 0.309859 0.2875 0.0156178 3.15481 2024-04-16T07:46:06.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 25 16 135 119 0.297464 0 - 3.15481 2024-04-16T07:46:06.762 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:06.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:06.883+0000 7f706626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:06.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:06.889+0000 7f706626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:07.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 26 16 135 119 0.286022 0 - 3.15481 2024-04-16T07:46:08.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 27 16 135 119 0.275429 0 - 3.15481 2024-04-16T07:46:08.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:09.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:09.111+0000 7f645ae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:09.117+0000 7f645ae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.190 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-04-16T07:46:09.190 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:09.191 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:46:09.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 28 16 135 119 0.265592 0 - 3.15481 2024-04-16T07:46:09.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:09.300+0000 7fc8d3217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:09.308+0000 7fc8d3217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:09.495 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:46:09.496 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:46:09.507 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-16T07:46:09.042909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8389152,"num_objects":121,"num_object_clones":0,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":256,"num_write_kb":9510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":109456,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377377904,"statfs":{"total":386547056640,"available":386434973696,"internally_reserved":0,"allocated":2080768,"data_stored":20301138,"data_compressed":957601,"data_compressed_allocated":1265664,"data_compressed_original":20004864,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.807482"},"pg_stats":[{"pgid":"3.f","version":"23'10","reported_seq":47,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.747929+0000","last_change":"2024-04-16T07:46:04.729472+0000","last_active":"2024-04-16T07:46:04.747929+0000","last_peered":"2024-04-16T07:46:04.747929+0000","last_clean":"2024-04-16T07:46:04.747929+0000","last_became_active":"2024-04-16T07:46:04.374237+0000","last_became_peered":"2024-04-16T07:46:04.374237+0000","last_unstale":"2024-04-16T07:46:04.747929+0000","last_undegraded":"2024-04-16T07:46:04.747929+0000","last_fullsized":"2024-04-16T07:46:04.747929+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'7","reported_seq":42,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.735449+0000","last_change":"2024-04-16T07:46:04.351186+0000","last_active":"2024-04-16T07:46:04.735449+0000","last_peered":"2024-04-16T07:46:04.735449+0000","last_clean":"2024-04-16T07:46:04.735449+0000","last_became_active":"2024-04-16T07:46:04.344770+0000","last_became_peered":"2024-04-16T07:46:04.344770+0000","last_unstale":"2024-04-16T07:46:04.735449+0000","last_undegraded":"2024-04-16T07:46:04.735449+0000","last_fullsized":"2024-04-16T07:46:04.735449+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524288,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"19'2","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.427136+0000","last_change":"2024-04-16T07:46:04.428284+0000","last_active":"2024-04-16T07:46:04.427136+0000","last_peered":"2024-04-16T07:46:04.427136+0000","last_clean":"2024-04-16T07:45:49.564881+0000","last_became_active":"2024-04-16T07:46:04.416295+0000","last_became_peered":"2024-04-16T07:46:04.416295+0000","last_unstale":"2024-04-16T07:46:04.427136+0000","last_undegraded":"2024-04-16T07:46:04.427136+0000","last_fullsized":"2024-04-16T07:46:04.427136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'3","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.767623+0000","last_change":"2024-04-16T07:45:49.395682+0000","last_active":"2024-04-16T07:46:04.767623+0000","last_peered":"2024-04-16T07:46:04.767623+0000","last_clean":"2024-04-16T07:46:04.767623+0000","last_became_active":"2024-04-16T07:45:49.381919+0000","last_became_peered":"2024-04-16T07:45:49.381919+0000","last_unstale":"2024-04-16T07:46:04.767623+0000","last_undegraded":"2024-04-16T07:46:04.767623+0000","last_fullsized":"2024-04-16T07:46:04.767623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'8","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.558704+0000","last_change":"2024-04-16T07:45:49.412308+0000","last_active":"2024-04-16T07:46:04.558704+0000","last_peered":"2024-04-16T07:46:04.558704+0000","last_clean":"2024-04-16T07:46:04.558704+0000","last_became_active":"2024-04-16T07:45:49.407698+0000","last_became_peered":"2024-04-16T07:45:49.407698+0000","last_unstale":"2024-04-16T07:46:04.558704+0000","last_undegraded":"2024-04-16T07:46:04.558704+0000","last_fullsized":"2024-04-16T07:46:04.558704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'9","reported_seq":44,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.872693+0000","last_change":"2024-04-16T07:46:04.716816+0000","last_active":"2024-04-16T07:46:04.872693+0000","last_peered":"2024-04-16T07:46:04.872693+0000","last_clean":"2024-04-16T07:46:04.872693+0000","last_became_active":"2024-04-16T07:46:04.383201+0000","last_became_peered":"2024-04-16T07:46:04.383201+0000","last_unstale":"2024-04-16T07:46:04.872693+0000","last_undegraded":"2024-04-16T07:46:04.872693+0000","last_fullsized":"2024-04-16T07:46:04.872693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'8","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.669720+0000","last_change":"2024-04-16T07:46:04.649565+0000","last_active":"2024-04-16T07:46:04.669720+0000","last_peered":"2024-04-16T07:46:04.669720+0000","last_clean":"2024-04-16T07:46:04.669720+0000","last_became_active":"2024-04-16T07:46:04.366049+0000","last_became_peered":"2024-04-16T07:46:04.366049+0000","last_unstale":"2024-04-16T07:46:04.669720+0000","last_undegraded":"2024-04-16T07:46:04.669720+0000","last_fullsized":"2024-04-16T07:46:04.669720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"19'4","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.401798+0000","last_change":"2024-04-16T07:46:04.403696+0000","last_active":"2024-04-16T07:46:04.401798+0000","last_peered":"2024-04-16T07:46:04.401798+0000","last_clean":"2024-04-16T07:45:49.740960+0000","last_became_active":"2024-04-16T07:46:04.393279+0000","last_became_peered":"2024-04-16T07:46:04.393279+0000","last_unstale":"2024-04-16T07:46:04.401798+0000","last_undegraded":"2024-04-16T07:46:04.401798+0000","last_fullsized":"2024-04-16T07:46:04.401798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":171,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.261917+0000","last_change":"2024-04-16T07:45:49.367607+0000","last_active":"2024-04-16T07:46:04.261917+0000","last_peered":"2024-04-16T07:46:04.261917+0000","last_clean":"2024-04-16T07:46:04.261917+0000","last_became_active":"2024-04-16T07:45:49.356206+0000","last_became_peered":"2024-04-16T07:45:49.356206+0000","last_unstale":"2024-04-16T07:46:04.261917+0000","last_undegraded":"2024-04-16T07:46:04.261917+0000","last_fullsized":"2024-04-16T07:46:04.261917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.404279+0000","last_change":"2024-04-16T07:46:04.406008+0000","last_active":"2024-04-16T07:46:04.404279+0000","last_peered":"2024-04-16T07:46:04.404279+0000","last_clean":"2024-04-16T07:45:49.321606+0000","last_became_active":"2024-04-16T07:46:04.390205+0000","last_became_peered":"2024-04-16T07:46:04.390205+0000","last_unstale":"2024-04-16T07:46:04.404279+0000","last_undegraded":"2024-04-16T07:46:04.404279+0000","last_fullsized":"2024-04-16T07:46:04.404279+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"19'3","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.378663+0000","last_change":"2024-04-16T07:46:04.379984+0000","last_active":"2024-04-16T07:46:04.378663+0000","last_peered":"2024-04-16T07:46:04.378663+0000","last_clean":"2024-04-16T07:45:49.692689+0000","last_became_active":"2024-04-16T07:46:04.370699+0000","last_became_peered":"2024-04-16T07:46:04.370699+0000","last_unstale":"2024-04-16T07:46:04.378663+0000","last_undegraded":"2024-04-16T07:46:04.378663+0000","last_fullsized":"2024-04-16T07:46:04.378663+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.404304+0000","last_change":"2024-04-16T07:46:04.406008+0000","last_active":"2024-04-16T07:46:04.404304+0000","last_peered":"2024-04-16T07:46:04.404304+0000","last_clean":"2024-04-16T07:45:52.611473+0000","last_became_active":"2024-04-16T07:46:04.394659+0000","last_became_peered":"2024-04-16T07:46:04.394659+0000","last_unstale":"2024-04-16T07:46:04.404304+0000","last_undegraded":"2024-04-16T07:46:04.404304+0000","last_fullsized":"2024-04-16T07:46:04.404304+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'8","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.909935+0000","last_change":"2024-04-16T07:46:04.341047+0000","last_active":"2024-04-16T07:46:04.909935+0000","last_peered":"2024-04-16T07:46:04.909935+0000","last_clean":"2024-04-16T07:46:04.909935+0000","last_became_active":"2024-04-16T07:46:04.334670+0000","last_became_peered":"2024-04-16T07:46:04.334670+0000","last_unstale":"2024-04-16T07:46:04.909935+0000","last_undegraded":"2024-04-16T07:46:04.909935+0000","last_fullsized":"2024-04-16T07:46:04.909935+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.431048+0000","last_change":"2024-04-16T07:46:04.432187+0000","last_active":"2024-04-16T07:46:04.431048+0000","last_peered":"2024-04-16T07:46:04.431048+0000","last_clean":"2024-04-16T07:45:34.144570+0000","last_became_active":"2024-04-16T07:46:04.424102+0000","last_became_peered":"2024-04-16T07:46:04.424102+0000","last_unstale":"2024-04-16T07:46:04.431048+0000","last_undegraded":"2024-04-16T07:46:04.431048+0000","last_fullsized":"2024-04-16T07:46:04.431048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'7","reported_seq":42,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.928816+0000","last_change":"2024-04-16T07:46:04.393460+0000","last_active":"2024-04-16T07:46:04.928816+0000","last_peered":"2024-04-16T07:46:04.928816+0000","last_clean":"2024-04-16T07:46:04.928816+0000","last_became_active":"2024-04-16T07:46:04.385804+0000","last_became_peered":"2024-04-16T07:46:04.385804+0000","last_unstale":"2024-04-16T07:46:04.928816+0000","last_undegraded":"2024-04-16T07:46:04.928816+0000","last_fullsized":"2024-04-16T07:46:04.928816+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524288,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.360068+0000","last_change":"2024-04-16T07:46:04.361779+0000","last_active":"2024-04-16T07:46:04.360068+0000","last_peered":"2024-04-16T07:46:04.360068+0000","last_clean":"2024-04-16T07:45:52.597297+0000","last_became_active":"2024-04-16T07:46:04.354553+0000","last_became_peered":"2024-04-16T07:46:04.354553+0000","last_unstale":"2024-04-16T07:46:04.360068+0000","last_undegraded":"2024-04-16T07:46:04.360068+0000","last_fullsized":"2024-04-16T07:46:04.360068+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.894075+0000","last_change":"2024-04-16T07:45:49.358481+0000","last_active":"2024-04-16T07:46:04.894075+0000","last_peered":"2024-04-16T07:46:04.894075+0000","last_clean":"2024-04-16T07:46:04.894075+0000","last_became_active":"2024-04-16T07:45:49.350849+0000","last_became_peered":"2024-04-16T07:45:49.350849+0000","last_unstale":"2024-04-16T07:46:04.894075+0000","last_undegraded":"2024-04-16T07:46:04.894075+0000","last_fullsized":"2024-04-16T07:46:04.894075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"19'4","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.408889+0000","last_change":"2024-04-16T07:46:04.410725+0000","last_active":"2024-04-16T07:46:04.408889+0000","last_peered":"2024-04-16T07:46:04.408889+0000","last_clean":"2024-04-16T07:45:49.724983+0000","last_became_active":"2024-04-16T07:46:04.399945+0000","last_became_peered":"2024-04-16T07:46:04.399945+0000","last_unstale":"2024-04-16T07:46:04.408889+0000","last_undegraded":"2024-04-16T07:46:04.408889+0000","last_fullsized":"2024-04-16T07:46:04.408889+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.276951+0000","last_change":"2024-04-16T07:45:49.453266+0000","last_active":"2024-04-16T07:46:04.276951+0000","last_peered":"2024-04-16T07:46:04.276951+0000","last_clean":"2024-04-16T07:46:04.276951+0000","last_became_active":"2024-04-16T07:45:49.447739+0000","last_became_peered":"2024-04-16T07:45:49.447739+0000","last_unstale":"2024-04-16T07:46:04.276951+0000","last_undegraded":"2024-04-16T07:46:04.276951+0000","last_fullsized":"2024-04-16T07:46:04.276951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'6","reported_seq":32,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.271432+0000","last_change":"2024-04-16T07:45:49.410057+0000","last_active":"2024-04-16T07:46:04.271432+0000","last_peered":"2024-04-16T07:46:04.271432+0000","last_clean":"2024-04-16T07:46:04.271432+0000","last_became_active":"2024-04-16T07:45:49.398629+0000","last_became_peered":"2024-04-16T07:45:49.398629+0000","last_unstale":"2024-04-16T07:46:04.271432+0000","last_undegraded":"2024-04-16T07:46:04.271432+0000","last_fullsized":"2024-04-16T07:46:04.271432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.272275+0000","last_change":"2024-04-16T07:45:49.370144+0000","last_active":"2024-04-16T07:46:04.272275+0000","last_peered":"2024-04-16T07:46:04.272275+0000","last_clean":"2024-04-16T07:46:04.272275+0000","last_became_active":"2024-04-16T07:45:49.363021+0000","last_became_peered":"2024-04-16T07:45:49.363021+0000","last_unstale":"2024-04-16T07:46:04.272275+0000","last_undegraded":"2024-04-16T07:46:04.272275+0000","last_fullsized":"2024-04-16T07:46:04.272275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'12","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.793273+0000","last_change":"2024-04-16T07:46:04.390430+0000","last_active":"2024-04-16T07:46:04.793273+0000","last_peered":"2024-04-16T07:46:04.793273+0000","last_clean":"2024-04-16T07:46:04.793273+0000","last_became_active":"2024-04-16T07:46:04.380096+0000","last_became_peered":"2024-04-16T07:46:04.380096+0000","last_unstale":"2024-04-16T07:46:04.793273+0000","last_undegraded":"2024-04-16T07:46:04.793273+0000","last_fullsized":"2024-04-16T07:46:04.793273+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.284073+0000","last_change":"2024-04-16T07:45:49.351498+0000","last_active":"2024-04-16T07:46:04.284073+0000","last_peered":"2024-04-16T07:46:04.284073+0000","last_clean":"2024-04-16T07:46:04.284073+0000","last_became_active":"2024-04-16T07:45:49.346290+0000","last_became_peered":"2024-04-16T07:45:49.346290+0000","last_unstale":"2024-04-16T07:46:04.284073+0000","last_undegraded":"2024-04-16T07:46:04.284073+0000","last_fullsized":"2024-04-16T07:46:04.284073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'6","reported_seq":41,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.536465+0000","last_change":"2024-04-16T07:46:04.366088+0000","last_active":"2024-04-16T07:46:04.536465+0000","last_peered":"2024-04-16T07:46:04.536465+0000","last_clean":"2024-04-16T07:46:04.536465+0000","last_became_active":"2024-04-16T07:46:04.359384+0000","last_became_peered":"2024-04-16T07:46:04.359384+0000","last_unstale":"2024-04-16T07:46:04.536465+0000","last_undegraded":"2024-04-16T07:46:04.536465+0000","last_fullsized":"2024-04-16T07:46:04.536465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:39.176594+0000","last_clean_scrub_stamp":"2024-04-16T07:45:39.176594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-16T07:46:04.397616+0000","last_change":"2024-04-16T07:46:04.399340+0000","last_active":"2024-04-16T07:46:04.397616+0000","last_peered":"2024-04-16T07:46:04.397616+0000","last_clean":"2024-04-16T07:45:49.416154+0000","last_became_active":"2024-04-16T07:46:04.386023+0000","last_became_peered":"2024-04-16T07:46:04.386023+0000","last_unstale":"2024-04-16T07:46:04.397616+0000","last_undegraded":"2024-04-16T07:46:04.397616+0000","last_fullsized":"2024-04-16T07:46:04.397616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":7798784,"num_objects":119,"num_object_clones":0,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":7616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":475136,"data_stored":4320122,"data_compressed":204117,"data_compressed_allocated":270336,"data_compressed_original":4243456,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27252,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":438272,"data_stored":3730298,"data_compressed":176130,"data_compressed_allocated":233472,"data_compressed_original":3653632,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27428,"kb_used_data":540,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":552960,"data_stored":5830175,"data_compressed":273692,"data_compressed_allocated":360448,"data_compressed_original":5767168,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411347,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27488,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":614400,"data_stored":6420543,"data_compressed":303662,"data_compressed_allocated":401408,"data_compressed_original":6340608,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:46:10.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 29 16 135 119 0.256433 0 - 3.15481 2024-04-16T07:46:11.220 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:11.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 30 16 135 119 0.247886 0 - 3.15481 2024-04-16T07:46:11.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:11.340+0000 7fc11f46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:11.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:11.346+0000 7fc11f46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:12.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 31 16 135 119 0.239889 0 - 3.15481 2024-04-16T07:46:13.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 32 16 135 119 0.232392 0 - 3.15481 2024-04-16T07:46:13.445 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:13.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:13.566+0000 7f188dead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:13.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:13.571+0000 7f188dead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 33 16 135 119 0.22535 0 - 3.15481 2024-04-16T07:46:14.509 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-04-16T07:46:14.509 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:14.509 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T07:46:14.510 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T07:46:14.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:14.621+0000 7f896ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:14.630+0000 7f896ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.645 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:14.645 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:14.645 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:14.645 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:14.645 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:14.645 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:14.645 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:14.646 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:14.646 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:14.646 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:14.646 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:14.646 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:14.649 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:14.649 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T07:46:14.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:14.758+0000 7f57ff806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:14.767+0000 7f57ff806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.781 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:14.781 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:14.781 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:14.781 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:14.782 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:14.782 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:14.782 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:14.782 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:14.782 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:14.782 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:14.782 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:14.782 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:14.784 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:14.785 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T07:46:14.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:14.893+0000 7f4dde26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:14.902+0000 7f4dde26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:14.920 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:14.920 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:14.920 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:14.920 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:14.921 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:14.921 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:14.921 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:14.921 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:14.921 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:14.921 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:14.921 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:14.921 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:14.923 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:14.924 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T07:46:15.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:15.032+0000 7f762e62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:15.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:15.041+0000 7f762e62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:15.058 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:15.058 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:15.058 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:15.058 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:15.059 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:15.059 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:15.059 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:15.059 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:15.059 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:15.059 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:15.059 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:15.059 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:15.061 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:15.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 34 16 135 119 0.218722 0 - 3.15481 2024-04-16T07:46:15.670 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:15.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:15.782+0000 7f1626d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:15.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:15.788+0000 7f1626d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:16.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 35 16 135 119 0.212473 0 - 3.15481 2024-04-16T07:46:17.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 36 16 329 313 0.543336 1.01042 0.0495954 1.83805 2024-04-16T07:46:17.889 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:18.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:18.059+0000 7f8c5ed9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:18.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:18.067+0000 7f8c5ed9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:18.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 37 16 551 535 0.903605 13.875 0.0561556 1.10559 2024-04-16T07:46:19.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 38 16 780 764 1.25643 14.3125 0.0675198 0.795208 2024-04-16T07:46:20.062 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-04-16T07:46:20.062 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:20.063 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T07:46:20.063 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T07:46:20.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:20.207+0000 7f8e50b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:20.220+0000 7f8e50b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 39 16 1007 991 1.58795 14.1875 0.0714648 0.629189 2024-04-16T07:46:20.330 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:20.330 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:20.330 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:20.330 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:20.330 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:20.330 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:20.330 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:20.331 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:20.331 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:20.331 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:20.331 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:20.331 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:20.335 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:20.335 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T07:46:20.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:20.484+0000 7f753706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:20.496+0000 7f753706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.568 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:20.608 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:20.611 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:20.611 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T07:46:20.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:20.742+0000 7f9a7e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:20.756+0000 7f9a7e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:20.817+0000 7f66af86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:20.830+0000 7f66af86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:20.857 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:20.858 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:20.858 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:20.858 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:20.858 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:20.858 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:20.858 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:20.858 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:20.858 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:20.858 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:20.859 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:20.859 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:20.860 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:20.861 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T07:46:21.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:21.005+0000 7f3adbe0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:21.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:21.017+0000 7f3adbe0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:21.050 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:21.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:46:21.284609+0000 min lat: 0.0147417 max lat: 23.2787 avg lat: 0.538404 2024-04-16T07:46:21.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:46:21.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 40 16 1204 1188 1.85603 12.3125 0.0380697 0.538404 2024-04-16T07:46:22.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 41 16 1413 1397 2.12932 13.0625 0.0908371 0.469256 2024-04-16T07:46:23.123 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:23.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 42 16 1628 1612 2.39853 13.4375 0.093497 0.416578 2024-04-16T07:46:23.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:23.281+0000 7f9da706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:23.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:23.290+0000 7f9da706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:24.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 43 16 1840 1824 2.65085 13.25 0.0975965 0.37689 2024-04-16T07:46:25.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 44 16 2057 2041 2.89881 13.5625 0.0302241 0.344646 2024-04-16T07:46:25.834 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:25.976+0000 7f4b4d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:25.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:25.985+0000 7f4b4d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.051 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-04-16T07:46:26.052 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:26.052 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T07:46:26.052 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T07:46:26.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:26.204+0000 7f1e3382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:26.217+0000 7f1e3382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 45 16 2258 2242 3.11353 12.5625 0.0575182 0.320981 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:26.331 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:26.332 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:26.332 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:26.336 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:26.336 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T07:46:26.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:26.507+0000 7f646fead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:26.522+0000 7f646fead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:26.658 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:26.659 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:26.662 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:26.662 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T07:46:26.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:26.817+0000 7f8beb95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:26.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:26.832+0000 7f8beb95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:27.004 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:27.004 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:27.005 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:27.009 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:27.009 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:27.010 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T07:46:27.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:27.186+0000 7fbb7baad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:27.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:27.201+0000 7fbb7baad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:27.259 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:46:27.259 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:46:27.259 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:46:27.260 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:46:27.260 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:46:27.260 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:46:27.260 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:46:27.260 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:46:27.260 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:46:27.260 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:46:27.260 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:46:27.260 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:46:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:27.263 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:46:27.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 46 16 2461 2445 3.32163 12.6875 0.110943 0.300777 2024-04-16T07:46:28.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 47 16 2672 2656 3.53151 13.1875 0.0521952 0.282967 2024-04-16T07:46:28.475 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:28.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:28.650+0000 7f868639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:28.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:28.662+0000 7f868639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:29.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 48 16 2881 2865 3.73004 13.0625 0.0993886 0.267826 2024-04-16T07:46:30.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 49 16 3085 3069 3.91409 12.75 0.046885 0.255221 2024-04-16T07:46:30.996 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:31.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:31.163+0000 7faad59bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:31.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:31.173+0000 7faad59bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:31.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 50 15 3299 3284 4.10453 13.4375 0.0737974 0.243471 2024-04-16T07:46:32.264 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-04-16T07:46:32.264 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:32.264 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [2, 3, 0, 1] 2024-04-16T07:46:32.264 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T07:46:32.264 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:32.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 51 16 3505 3489 4.27525 12.8125 0.0917238 0.233716 2024-04-16T07:46:32.334 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T07:46:33.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 52 16 3514 3498 4.20385 0.5625 0.0287526 0.233341 2024-04-16T07:46:33.787 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:33.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:33.910+0000 7f763dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:33.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:33.915+0000 7f763dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:34.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 53 16 3514 3498 4.12453 0 - 0.233341 2024-04-16T07:46:35.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 54 16 3514 3498 4.04815 0 - 0.233341 2024-04-16T07:46:36.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 55 16 3514 3498 3.97455 0 - 0.233341 2024-04-16T07:46:37.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 56 16 3514 3498 3.90357 0 - 0.233341 2024-04-16T07:46:37.335 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [3, 0, 1] 2024-04-16T07:46:37.335 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:37.335 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-16T07:46:37.335 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T07:46:37.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T07:46:37.338 INFO:tasks.ceph.osd.2:Started 2024-04-16T07:46:37.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T07:46:37.459 INFO:teuthology.orchestra.run.smithi167.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:46:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:37.463 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T07:46:38.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 57 16 3514 3498 3.83509 0 - 0.233341 2024-04-16T07:46:39.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 58 16 3514 3498 3.76896 0 - 0.233341 2024-04-16T07:46:40.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 59 16 3514 3498 3.70508 0 - 0.233341 2024-04-16T07:46:41.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:46:41.286741+0000 min lat: 0.0147417 max lat: 23.2787 avg lat: 0.233341 2024-04-16T07:46:41.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:46:41.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 60 16 3514 3498 3.64333 0 - 0.233341 2024-04-16T07:46:42.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 61 16 3514 3498 3.5836 0 - 0.233341 2024-04-16T07:46:42.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T07:46:42.571 INFO:teuthology.orchestra.run.smithi167.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:46:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:42.575 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-16T07:46:43.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 62 16 3514 3498 3.5258 0 - 0.233341 2024-04-16T07:46:44.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 63 16 3514 3498 3.46983 0 - 0.233341 2024-04-16T07:46:44.737 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-16T07:46:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:46:44.868 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:46:45.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 64 16 3514 3498 3.41562 0 - 0.233341 2024-04-16T07:46:46.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 65 16 3514 3498 3.36307 0 - 0.233341 2024-04-16T07:46:46.869 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:47.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:47.017+0000 7f9ad262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:47.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:47.024+0000 7f9ad262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:47.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 66 16 3551 3535 3.34715 0.165179 0.0723913 0.290054 2024-04-16T07:46:47.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "ops_in_flight": [ 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 295, 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.694448+0000" 2024-04-16T07:46:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.694464+0000" 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.694766+0000" 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.695156+0000" 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.695446+0000" 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 93, 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296744+0000" 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296752+0000" 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296902+0000" 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296992+0000" 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.297133+0000" 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.151696+0000" 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 207, 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.153890+0000" 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.153901+0000" 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.154530+0000" 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.154729+0000" 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.155856+0000" 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.260414+0000" 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 219, 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.217136+0000" 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.217156+0000" 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.218991+0000" 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.219518+0000" 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.219731+0000" 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.297936+0000" 2024-04-16T07:46:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 239, 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.332252+0000" 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.332268+0000" 2024-04-16T07:46:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.332604+0000" 2024-04-16T07:46:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.332883+0000" 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.334074+0000" 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.402782+0000" 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 232, 2024-04-16T07:46:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.299619+0000" 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.299645+0000" 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.299929+0000" 2024-04-16T07:46:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.300131+0000" 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.300310+0000" 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.410160+0000" 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 245, 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.371425+0000" 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.371437+0000" 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.373671+0000" 2024-04-16T07:46:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.374353+0000" 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.375376+0000" 2024-04-16T07:46:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.441814+0000" 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 251, 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.405130+0000" 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.405140+0000" 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.405435+0000" 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.405561+0000" 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.405677+0000" 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.528430+0000" 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 256, 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.436379+0000" 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.436398+0000" 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.437093+0000" 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.437412+0000" 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.437684+0000" 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.544285+0000" 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 258, 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.448314+0000" 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.448324+0000" 2024-04-16T07:46:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.449606+0000" 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.450047+0000" 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.450837+0000" 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.564473+0000" 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 260, 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.463981+0000" 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.463991+0000" 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.467978+0000" 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.473216+0000" 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.475145+0000" 2024-04-16T07:46:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.581267+0000" 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 263, 2024-04-16T07:46:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.479777+0000" 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.479801+0000" 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.480342+0000" 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.480991+0000" 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.482202+0000" 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.593753+0000" 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 266, 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.495626+0000" 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.495649+0000" 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.497279+0000" 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.499437+0000" 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.500639+0000" 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.608644+0000" 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 278, 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.586154+0000" 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.586165+0000" 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.586336+0000" 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.586576+0000" 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.586965+0000" 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.687430+0000" 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 285, 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.636939+0000" 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.636949+0000" 2024-04-16T07:46:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.637326+0000" 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.637964+0000" 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.638366+0000" 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.738746+0000" 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 287, 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.660463+0000" 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.660482+0000" 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.661051+0000" 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.661234+0000" 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.662681+0000" 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.743540+0000" 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 289, 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.673082+0000" 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.673092+0000" 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.673641+0000" 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.676849+0000" 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.680692+0000" 2024-04-16T07:46:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 299, 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.739548+0000" 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.739559+0000" 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.739793+0000" 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.740233+0000" 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.742634+0000" 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "replicated_request", 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 300, 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "RepRequest": { 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "reqid": "client.4221.0:3626", 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "pgid": "3.0", 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "map_epoch": 26, 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "min_epoch": 25, 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "oid": "3:0f3372ed:::benchmark_data_smithi079_31310_object3625:head", 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "from": "3" 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 88, 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.293122+0000" 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.293151+0000" 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.293320+0000" 2024-04-16T07:46:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.293422+0000" 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.293796+0000" 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.973522+0000" 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 91, 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.295385+0000" 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.295392+0000" 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.295537+0000" 2024-04-16T07:46:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.295644+0000" 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296502+0000" 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.181484+0000" 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 92, 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296146+0000" 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296154+0000" 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296288+0000" 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296466+0000" 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.297199+0000" 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.830 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.831 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.831 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.831 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.184718+0000" 2024-04-16T07:46:47.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 182, 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.066694+0000" 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.066734+0000" 2024-04-16T07:46:47.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.067617+0000" 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.068398+0000" 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.833 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.073947+0000" 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.250348+0000" 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 240, 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.337820+0000" 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.337832+0000" 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.835 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.337994+0000" 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.338159+0000" 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.836 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.341513+0000" 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.406003+0000" 2024-04-16T07:46:47.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 248, 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.381837+0000" 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.838 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.381847+0000" 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.383781+0000" 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.839 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.384458+0000" 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.385966+0000" 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.465577+0000" 2024-04-16T07:46:47.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 273, 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.548953+0000" 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.548963+0000" 2024-04-16T07:46:47.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.553681+0000" 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.554994+0000" 2024-04-16T07:46:47.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.558362+0000" 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.636647+0000" 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 276, 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.578923+0000" 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.578952+0000" 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.844 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.579798+0000" 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.580495+0000" 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.845 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.582533+0000" 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.664588+0000" 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 279, 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.610797+0000" 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.610820+0000" 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.847 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.611103+0000" 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.611275+0000" 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.612412+0000" 2024-04-16T07:46:47.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.709890+0000" 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.849 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 292, 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.690912+0000" 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.690928+0000" 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.691254+0000" 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.850 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.692330+0000" 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.694756+0000" 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.851 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 89, 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.852 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.294006+0000" 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.294014+0000" 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.294157+0000" 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.853 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.294293+0000" 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.295230+0000" 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.854 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.200644+0000" 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 90, 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.855 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.294777+0000" 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.294785+0000" 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.294954+0000" 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.856 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.295069+0000" 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:46.296012+0000" 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.283292+0000" 2024-04-16T07:46:47.857 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 176, 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.049716+0000" 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.858 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.049731+0000" 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.050587+0000" 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.859 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.051823+0000" 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.054551+0000" 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.283780+0000" 2024-04-16T07:46:47.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 269, 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.524438+0000" 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.524465+0000" 2024-04-16T07:46:47.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.526966+0000" 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.527501+0000" 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.862 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.534813+0000" 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.612877+0000" 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 270, 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.537748+0000" 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.864 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.537772+0000" 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.539814+0000" 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.540053+0000" 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.865 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.542750+0000" 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.631162+0000" 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "client_request", 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 275, 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.565542+0000" 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.565564+0000" 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.867 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.567697+0000" 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.568706+0000" 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.573124+0000" 2024-04-16T07:46:47.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: "blocking_event": { 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time_event": { 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:46:47.652976+0000" 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-16T07:46:47.869 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-16T07:46:48.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 67 16 3744 3728 3.47721 12.0625 0.125031 0.287426 2024-04-16T07:46:49.281 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:49.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 68 16 3952 3936 3.61723 13 0.114999 0.276284 2024-04-16T07:46:49.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:49.425+0000 7f955ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:49.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:49.434+0000 7f955ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:50.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 69 16 4146 4130 3.74051 12.125 0.0825887 0.267131 2024-04-16T07:46:51.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 70 16 4352 4336 3.87099 12.875 0.0968313 0.258139 2024-04-16T07:46:51.854 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:46:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:52.027+0000 7ff7e6d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:52.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:52.036+0000 7ff7e6d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:52.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 71 16 4539 4523 3.98106 11.6875 0.0961036 0.250958 2024-04-16T07:46:52.870 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 0, 1, 2] 2024-04-16T07:46:52.871 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:52.871 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [3, 0, 1, 2] 2024-04-16T07:46:52.871 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T07:46:52.871 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:46:52.941 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T07:46:53.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 72 16 4659 4643 4.02992 7.5 0.0220849 0.246834 2024-04-16T07:46:54.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 73 16 4659 4643 3.97471 0 - 0.246834 2024-04-16T07:46:54.452 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:46:54.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:54.573+0000 7faba2a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:54.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:46:54.579+0000 7faba2a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:46:55.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 74 16 4659 4643 3.921 0 - 0.246834 2024-04-16T07:46:56.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 75 16 4659 4643 3.86872 0 - 0.246834 2024-04-16T07:46:57.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 76 16 4659 4643 3.81781 0 - 0.246834 2024-04-16T07:46:57.942 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [3, 0, 2] 2024-04-16T07:46:57.942 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:46:57.942 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-16T07:46:57.942 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:46:57.942 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:46:57.946 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:46:57.946 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:46:58.060 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:46:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:46:58.063 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:46:58.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 77 16 4659 4643 3.76823 0 - 0.246834 2024-04-16T07:46:59.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 78 16 4659 4643 3.71992 0 - 0.246834 2024-04-16T07:47:00.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 79 16 4659 4643 3.67283 0 - 0.246834 2024-04-16T07:47:01.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:47:01.289151+0000 min lat: 0.0147417 max lat: 23.2787 avg lat: 0.246834 2024-04-16T07:47:01.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:47:01.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 80 16 4659 4643 3.62692 0 - 0.246834 2024-04-16T07:47:02.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 81 16 4659 4643 3.58214 0 - 0.246834 2024-04-16T07:47:03.065 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:47:03.171 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:03.175 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:47:03.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 82 16 4659 4643 3.53846 0 - 0.246834 2024-04-16T07:47:04.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 83 16 4659 4643 3.49583 0 - 0.246834 2024-04-16T07:47:05.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 84 16 4659 4643 3.45421 0 - 0.246834 2024-04-16T07:47:06.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 85 16 4659 4643 3.41357 0 - 0.246834 2024-04-16T07:47:07.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 86 16 4659 4643 3.37388 0 - 0.246834 2024-04-16T07:47:08.175 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:47:08.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 87 16 4659 4643 3.3351 0 - 0.246834 2024-04-16T07:47:08.290 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:08.295 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:47:09.117 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-16T07:47:09.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 88 16 4659 4643 3.29721 0 - 0.246834 2024-04-16T07:47:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:47:09.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:47:10.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 89 16 4659 4643 3.26016 0 - 0.246834 2024-04-16T07:47:11.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 90 16 4661 4645 3.22532 0.00694444 18.405 0.254649 2024-04-16T07:47:11.290 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:47:11.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:11.435+0000 7fedcff5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:11.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:11.442+0000 7fedcff5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:12.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Total time run: 90.2771 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Total writes made: 4661 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Write size: 65536 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Object size: 65536 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Bandwidth (MB/sec): 3.22687 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Stddev Bandwidth: 5.44565 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Max bandwidth (MB/sec): 14.3125 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Min bandwidth (MB/sec): 0 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Average IOPS: 51 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Stddev IOPS: 87.139 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Max IOPS: 229 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Min IOPS: 0 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Average Latency(s): 0.309545 2024-04-16T07:47:12.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Stddev Latency(s): 1.89322 2024-04-16T07:47:12.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Max latency(s): 23.2787 2024-04-16T07:47:12.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Min latency(s): 0.0147417 2024-04-16T07:47:12.311 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Cleaning up (deleting benchmark objects) 2024-04-16T07:47:13.296 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:47:13.468 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-16T07:47:13.468 INFO:teuthology.orchestra.run.smithi079.stdout: "ops_in_flight": [ 2024-04-16T07:47:13.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.468 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "replicated_request", 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 424, 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: "RepRequest": { 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: "reqid": "client.4221.0:4931", 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: "pgid": "3.6", 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 29, 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: "min_epoch": 28, 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: "oid": "3:676bacec:::benchmark_data_smithi079_31310_object267:head", 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: "from": "2" 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "replicated_request", 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 425, 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "RepRequest": { 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "reqid": "client.4221.0:4933", 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "pgid": "3.a", 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 29, 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "min_epoch": 28, 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "oid": "3:59eb0ff3:::benchmark_data_smithi079_31310_object269:head", 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: "from": "0" 2024-04-16T07:47:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 155, 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.364587+0000" 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.471 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.364601+0000" 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.366336+0000" 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.368927+0000" 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.472 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.372796+0000" 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.505033+0000" 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.473 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 169, 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.417349+0000" 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.417366+0000" 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.417943+0000" 2024-04-16T07:47:13.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.419038+0000" 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.419667+0000" 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.475 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.512878+0000" 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 211, 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314773+0000" 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.476 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314780+0000" 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314934+0000" 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.477 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.315040+0000" 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.315138+0000" 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.478 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.358533+0000" 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 229, 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.408803+0000" 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.479 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.408816+0000" 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.410686+0000" 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.411022+0000" 2024-04-16T07:47:13.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.411219+0000" 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.446744+0000" 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 250, 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.518068+0000" 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.518087+0000" 2024-04-16T07:47:13.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.518532+0000" 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.518782+0000" 2024-04-16T07:47:13.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.519007+0000" 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.554811+0000" 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.484 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 264, 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.608867+0000" 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.608882+0000" 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.609157+0000" 2024-04-16T07:47:13.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.609292+0000" 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.609385+0000" 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.654389+0000" 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 265, 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.618136+0000" 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.618147+0000" 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.618344+0000" 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.619563+0000" 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.619692+0000" 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.667119+0000" 2024-04-16T07:47:13.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 280, 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.692329+0000" 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.692347+0000" 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.490 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.692564+0000" 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.692656+0000" 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.692757+0000" 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.491 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.729103+0000" 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 291, 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.743829+0000" 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.492 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.743841+0000" 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.744022+0000" 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.744165+0000" 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.493 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.744452+0000" 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.783336+0000" 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 311, 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.843464+0000" 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.843493+0000" 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.495 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.843727+0000" 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.843953+0000" 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.844077+0000" 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.496 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.863004+0000" 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 318, 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.884064+0000" 2024-04-16T07:47:13.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.884075+0000" 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.884231+0000" 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.498 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.884329+0000" 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.884510+0000" 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.929154+0000" 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 322, 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.902533+0000" 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.902549+0000" 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.902688+0000" 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.902856+0000" 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.902972+0000" 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.976389+0000" 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 329, 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.943615+0000" 2024-04-16T07:47:13.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.943631+0000" 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.944566+0000" 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.503 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.944894+0000" 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.945904+0000" 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.013021+0000" 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 331, 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.948500+0000" 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.948515+0000" 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.949701+0000" 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.505 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.949986+0000" 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.954956+0000" 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.025092+0000" 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 343, 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.016859+0000" 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.507 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.016877+0000" 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.017618+0000" 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.019637+0000" 2024-04-16T07:47:13.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.022227+0000" 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.092886+0000" 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.509 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 344, 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.033244+0000" 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.033262+0000" 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.034360+0000" 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.510 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.034973+0000" 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.035812+0000" 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.511 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.103775+0000" 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 373, 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.174988+0000" 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.174999+0000" 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.512 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.175217+0000" 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.175393+0000" 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.175626+0000" 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.513 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.220681+0000" 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 422, 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.436872+0000" 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.514 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.436883+0000" 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.437068+0000" 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.437180+0000" 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.515 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.440256+0000" 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.516 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 161, 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.382552+0000" 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.382589+0000" 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.384707+0000" 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.384913+0000" 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.387172+0000" 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.476838+0000" 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 166, 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.410859+0000" 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.519 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.410875+0000" 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.411097+0000" 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.411347+0000" 2024-04-16T07:47:13.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.411894+0000" 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.496192+0000" 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 209, 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.313705+0000" 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.313714+0000" 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.522 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.313866+0000" 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314010+0000" 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314338+0000" 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.523 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.373088+0000" 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 212, 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.318069+0000" 2024-04-16T07:47:13.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.318081+0000" 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.318430+0000" 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.525 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.318565+0000" 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.323883+0000" 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.383539+0000" 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.526 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 219, 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.358298+0000" 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.358314+0000" 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.527 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.358872+0000" 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.359055+0000" 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.360291+0000" 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.528 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.408106+0000" 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 256, 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.529 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.550405+0000" 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.550422+0000" 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.552543+0000" 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.552965+0000" 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.554243+0000" 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.606966+0000" 2024-04-16T07:47:13.531 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 281, 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.699710+0000" 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.699719+0000" 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.532 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.700678+0000" 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.701295+0000" 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.533 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.705520+0000" 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.760737+0000" 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.534 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 289, 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.738782+0000" 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.738802+0000" 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.739178+0000" 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.739374+0000" 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.739985+0000" 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.536 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.821858+0000" 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 333, 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.969642+0000" 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.969654+0000" 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.537 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.972863+0000" 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.973981+0000" 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.538 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.975365+0000" 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.045519+0000" 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 352, 2024-04-16T07:47:13.539 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.075123+0000" 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.075141+0000" 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.076323+0000" 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.076581+0000" 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.076903+0000" 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.541 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.136950+0000" 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 364, 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.133184+0000" 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.133197+0000" 2024-04-16T07:47:13.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.133505+0000" 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.133674+0000" 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.134558+0000" 2024-04-16T07:47:13.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.202452+0000" 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 374, 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.187933+0000" 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.187951+0000" 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.188528+0000" 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.188750+0000" 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.189203+0000" 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.240345+0000" 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 394, 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.288166+0000" 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.288185+0000" 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.547 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.288605+0000" 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.288977+0000" 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.291781+0000" 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.548 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.353453+0000" 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 397, 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.293139+0000" 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.549 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.293148+0000" 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.293337+0000" 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.293467+0000" 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.550 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.299716+0000" 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.376430+0000" 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 409, 2024-04-16T07:47:13.551 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.365373+0000" 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.365400+0000" 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.365867+0000" 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.366105+0000" 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.367569+0000" 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.553 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.442304+0000" 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 417, 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.400630+0000" 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.554 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.400652+0000" 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.403312+0000" 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.404006+0000" 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.555 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.406280+0000" 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 152, 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.359252+0000" 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.359274+0000" 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.359775+0000" 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.557 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.360729+0000" 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.363594+0000" 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.366881+0000" 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 163, 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.394558+0000" 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.394586+0000" 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.559 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.396412+0000" 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.400048+0000" 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.405684+0000" 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:11.492792+0000" 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 207, 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.290926+0000" 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.561 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.290944+0000" 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.291151+0000" 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.291252+0000" 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.562 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.291572+0000" 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.306729+0000" 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 208, 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.563 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.307503+0000" 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.307531+0000" 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.307659+0000" 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.564 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.307802+0000" 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.308080+0000" 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.312517+0000" 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 210, 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314268+0000" 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314275+0000" 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.566 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314419+0000" 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314511+0000" 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.314926+0000" 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.567 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.374395+0000" 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 213, 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.323624+0000" 2024-04-16T07:47:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.323637+0000" 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.324125+0000" 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.569 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.324856+0000" 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.327983+0000" 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.391500+0000" 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.570 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 223, 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.381022+0000" 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.381038+0000" 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.571 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.381159+0000" 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.381283+0000" 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.381558+0000" 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.572 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.437386+0000" 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 233, 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.427393+0000" 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.573 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.427409+0000" 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.427868+0000" 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.428386+0000" 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.574 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.429605+0000" 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.465426+0000" 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.575 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 237, 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.444381+0000" 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.444416+0000" 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.444680+0000" 2024-04-16T07:47:13.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.444846+0000" 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.446322+0000" 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.577 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.483543+0000" 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 253, 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.532164+0000" 2024-04-16T07:47:13.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.532177+0000" 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.533593+0000" 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.533714+0000" 2024-04-16T07:47:13.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.535615+0000" 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.582700+0000" 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.580 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 259, 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.582307+0000" 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.582319+0000" 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.582760+0000" 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.581 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.583015+0000" 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.585651+0000" 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.607946+0000" 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 261, 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.586509+0000" 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.583 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.586523+0000" 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.586672+0000" 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.586799+0000" 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.584 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.587062+0000" 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.641827+0000" 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 272, 2024-04-16T07:47:13.585 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.656700+0000" 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.656711+0000" 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.657163+0000" 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.586 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.659953+0000" 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.662297+0000" 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.587 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.716125+0000" 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 267, 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.635687+0000" 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.588 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.635702+0000" 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.636454+0000" 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.636986+0000" 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.589 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.637520+0000" 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.716413+0000" 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.590 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 276, 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.675043+0000" 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.675061+0000" 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.676987+0000" 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.591 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.677327+0000" 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.677732+0000" 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.727590+0000" 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 278, 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.678682+0000" 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.593 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.678697+0000" 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.679605+0000" 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.594 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.679857+0000" 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.684586+0000" 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.735186+0000" 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.595 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 285, 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.721181+0000" 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.721205+0000" 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.596 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.721886+0000" 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.722080+0000" 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.724667+0000" 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.597 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.787097+0000" 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 300, 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.786285+0000" 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.598 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.786299+0000" 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.786546+0000" 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.786702+0000" 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.599 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.787383+0000" 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.840442+0000" 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 306, 2024-04-16T07:47:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.821267+0000" 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.821284+0000" 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.821524+0000" 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.601 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.821802+0000" 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.822341+0000" 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.602 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.859302+0000" 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 312, 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.853536+0000" 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.853566+0000" 2024-04-16T07:47:13.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.853795+0000" 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.853947+0000" 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.604 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.854309+0000" 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.898759+0000" 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 315, 2024-04-16T07:47:13.605 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.867620+0000" 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.867631+0000" 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.867918+0000" 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.868102+0000" 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.868464+0000" 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.607 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.918881+0000" 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 319, 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.889930+0000" 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.889941+0000" 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.608 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.890492+0000" 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.890714+0000" 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.894944+0000" 2024-04-16T07:47:13.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.950478+0000" 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 320, 2024-04-16T07:47:13.610 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.898136+0000" 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.898146+0000" 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.898674+0000" 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.611 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.898777+0000" 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.904870+0000" 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.612 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.968628+0000" 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 324, 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.921053+0000" 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.921064+0000" 2024-04-16T07:47:13.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.924705+0000" 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.925715+0000" 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.614 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.927707+0000" 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.996643+0000" 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.615 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 326, 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.928371+0000" 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.928380+0000" 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.928847+0000" 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.616 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.929719+0000" 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.933803+0000" 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.617 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.997437+0000" 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 328, 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.932332+0000" 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.932348+0000" 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.932660+0000" 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.932920+0000" 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.619 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.936636+0000" 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.005427+0000" 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.620 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 335, 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.980671+0000" 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.980683+0000" 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.981930+0000" 2024-04-16T07:47:13.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.983553+0000" 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:12.987148+0000" 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.622 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.041879+0000" 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 340, 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.623 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.013781+0000" 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.013792+0000" 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.014120+0000" 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.624 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.014306+0000" 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.014733+0000" 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.625 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.075557+0000" 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 348, 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.060564+0000" 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.626 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.060595+0000" 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.060895+0000" 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.061578+0000" 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.627 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.062376+0000" 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.145135+0000" 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 351, 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.070596+0000" 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.070612+0000" 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.071387+0000" 2024-04-16T07:47:13.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.072045+0000" 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.073045+0000" 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.630 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.151781+0000" 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 354, 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.631 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.088020+0000" 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.088042+0000" 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.090783+0000" 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.632 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.092340+0000" 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.093794+0000" 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.169891+0000" 2024-04-16T07:47:13.633 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 357, 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.100376+0000" 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.100394+0000" 2024-04-16T07:47:13.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.101809+0000" 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.101970+0000" 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.635 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.104182+0000" 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.176597+0000" 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 359, 2024-04-16T07:47:13.636 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.108829+0000" 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.108840+0000" 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.109118+0000" 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.637 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.110078+0000" 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.113172+0000" 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.184580+0000" 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 363, 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.125024+0000" 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.639 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.125057+0000" 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.127237+0000" 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.127472+0000" 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.640 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.130909+0000" 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.204858+0000" 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.641 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 367, 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.145950+0000" 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.145971+0000" 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.146437+0000" 2024-04-16T07:47:13.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.148091+0000" 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.149282+0000" 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.209936+0000" 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 371, 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.165464+0000" 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.644 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.165491+0000" 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.165744+0000" 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.165916+0000" 2024-04-16T07:47:13.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.166750+0000" 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.213336+0000" 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 381, 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.218071+0000" 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.218086+0000" 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.219241+0000" 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.647 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.219642+0000" 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.223244+0000" 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.283255+0000" 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 396, 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.292288+0000" 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.649 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.292307+0000" 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.292566+0000" 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.292800+0000" 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.650 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.293678+0000" 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.333880+0000" 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.651 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 399, 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.306623+0000" 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.306635+0000" 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.307324+0000" 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.307450+0000" 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.312708+0000" 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.389707+0000" 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.653 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 413, 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.388075+0000" 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.388087+0000" 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.654 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.388350+0000" 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.388599+0000" 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.389957+0000" 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:47:13.655 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:47:13.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:47:13.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:47:13.656 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:47:13.656 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:47:13.443822+0000" 2024-04-16T07:47:13.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:47:13.656 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-16T07:47:13.656 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-16T07:47:13.656 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:47:13.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:13.844+0000 7f2c9c6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:13.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:13.856+0000 7f2c9b48b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:16.278 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:47:16.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:16.430+0000 7f52dd7fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:16.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:16.440+0000 7f52dd7fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:18.658 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 0, 2, 1] 2024-04-16T07:47:18.658 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:47:18.659 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [3, 0, 2, 1] 2024-04-16T07:47:18.659 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T07:47:18.659 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:47:18.754 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T07:47:18.811 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:47:18.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:18.921+0000 7f6ab999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:18.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:18.927+0000 7f6ab999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:23.755 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [3, 2, 1] 2024-04-16T07:47:23.755 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:47:23.755 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-16T07:47:23.755 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:47:23.756 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:47:23.759 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:47:23.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:47:23.874 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:23.878 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-16T07:47:28.879 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:47:28.999 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-16T07:47:28.999 INFO:teuthology.orchestra.run.smithi079.stdout: "ops_in_flight": [] 2024-04-16T07:47:28.999 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-16T07:47:29.124 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:47:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:47:29.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:47:31.321 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:47:31.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:31.488+0000 7f01ea86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:31.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:31.497+0000 7f01ea86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:33.771 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:47:33.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:33.925+0000 7fbae6554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:33.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:33.932+0000 7fbae6554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:34.010 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 2, 1, 0] 2024-04-16T07:47:34.010 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:47:34.010 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [3, 2, 1, 0] 2024-04-16T07:47:34.011 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T07:47:34.011 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:47:34.100 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T07:47:39.100 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [2, 1, 0] 2024-04-16T07:47:39.101 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:47:39.101 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [2, 1, 0] 2024-04-16T07:47:39.101 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T07:47:39.101 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:47:39.181 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T07:47:44.182 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 0] live_osds: [2, 1] 2024-04-16T07:47:44.182 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:47:44.182 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-16T07:47:44.183 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T07:47:44.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T07:47:44.186 INFO:tasks.ceph.osd.3:Started 2024-04-16T07:47:44.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:44.302 INFO:teuthology.orchestra.run.smithi167.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:44.306 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:49.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:49.412 INFO:teuthology.orchestra.run.smithi167.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:49.415 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:47:53.742 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-16T07:47:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:47:53.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:47:54.416 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:47:54.526 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-16T07:47:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "ops_in_flight": [] 2024-04-16T07:47:54.526 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-16T07:47:55.748 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:47:55.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:55.906+0000 7fae8d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:55.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:47:55.911+0000 7fae8d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:47:59.536 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [2, 1, 3] 2024-04-16T07:47:59.536 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:47:59.536 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-16T07:47:59.536 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:47:59.536 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:47:59.539 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:47:59.539 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:47:59.653 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:47:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:47:59.656 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-16T07:48:04.126 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:48:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:48:04.334 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:48:04.657 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:48:04.765 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-16T07:48:04.765 INFO:teuthology.orchestra.run.smithi079.stdout: "ops_in_flight": [] 2024-04-16T07:48:04.765 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-16T07:48:06.335 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:48:06.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:06.476+0000 7fda1b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:06.483+0000 7fda1b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:08.842 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:48:08.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:08.985+0000 7f7527e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:08.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:08.992+0000 7f7527e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:09.775 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 1, 3, 0] 2024-04-16T07:48:09.775 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:48:09.775 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [2, 1, 3, 0] 2024-04-16T07:48:09.776 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T07:48:09.776 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:48:09.857 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T07:48:11.345 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:48:11.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:11.466+0000 7f6015c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:11.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:11.471+0000 7f6015c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:14.858 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [2, 1, 3] 2024-04-16T07:48:14.858 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:48:14.858 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-16T07:48:14.858 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:48:14.859 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:48:14.862 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:48:14.862 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:48:14.976 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:14.980 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-16T07:48:19.981 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:48:20.091 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:20.094 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-16T07:48:20.756 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:48:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:48:20.968 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:48:22.970 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:48:23.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:23.099+0000 7f49d379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:23.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:23.108+0000 7f49d379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:25.095 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-16T07:48:25.294 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-16T07:48:25.294 INFO:teuthology.orchestra.run.smithi079.stdout: "ops_in_flight": [ 2024-04-16T07:48:25.294 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.294 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 503, 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.203043+0000" 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.203075+0000" 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.203798+0000" 2024-04-16T07:48:25.295 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.204237+0000" 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.204390+0000" 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.296 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 504, 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.226528+0000" 2024-04-16T07:48:25.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.226541+0000" 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.226919+0000" 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.298 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.227234+0000" 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.227531+0000" 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.299 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 29, 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.768140+0000" 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.768172+0000" 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.300 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.768403+0000" 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.768621+0000" 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.855923+0000" 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.857243+0000" 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.301 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.937529+0000" 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 31, 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.302 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.769368+0000" 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.769376+0000" 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.769506+0000" 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.303 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.861209+0000" 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.862536+0000" 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.968334+0000" 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.304 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 53, 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.860692+0000" 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.860702+0000" 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.305 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.861297+0000" 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.865003+0000" 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.866590+0000" 2024-04-16T07:48:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.179279+0000" 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.307 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 32, 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.769680+0000" 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.769687+0000" 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.770526+0000" 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.308 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.861932+0000" 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.862714+0000" 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.309 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.183637+0000" 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 48, 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.829846+0000" 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.310 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.829860+0000" 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.844990+0000" 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.863129+0000" 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.311 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.864218+0000" 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.186924+0000" 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 133, 2024-04-16T07:48:25.312 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.914491+0000" 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.914508+0000" 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.916256+0000" 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.313 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.917418+0000" 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.918639+0000" 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.314 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.197623+0000" 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 182, 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.115254+0000" 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.115270+0000" 2024-04-16T07:48:25.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.115457+0000" 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.116420+0000" 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.119157+0000" 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.316 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.239589+0000" 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 290, 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.767564+0000" 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.317 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.767578+0000" 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.767795+0000" 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.318 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.767935+0000" 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.768098+0000" 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.813792+0000" 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 291, 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.783515+0000" 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.783532+0000" 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.786343+0000" 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.786739+0000" 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.787267+0000" 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.838316+0000" 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 313, 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.952823+0000" 2024-04-16T07:48:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.952841+0000" 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.953084+0000" 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.953297+0000" 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.953504+0000" 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.998604+0000" 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 315, 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.975309+0000" 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.975335+0000" 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.976425+0000" 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.976596+0000" 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.977101+0000" 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.025194+0000" 2024-04-16T07:48:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 318, 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.994615+0000" 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.994630+0000" 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.995143+0000" 2024-04-16T07:48:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.995504+0000" 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.995917+0000" 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.054375+0000" 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 352, 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.214596+0000" 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.214621+0000" 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.217634+0000" 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.217798+0000" 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.217972+0000" 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.267813+0000" 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 354, 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.221078+0000" 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.221088+0000" 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.223982+0000" 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.224623+0000" 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.224971+0000" 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.274505+0000" 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 372, 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.332280+0000" 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.332297+0000" 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.332460+0000" 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.332575+0000" 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.332677+0000" 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.382708+0000" 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 375, 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.354764+0000" 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.354780+0000" 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.355033+0000" 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.355194+0000" 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.355367+0000" 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.404834+0000" 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 390, 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.454157+0000" 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.454176+0000" 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.454413+0000" 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.454523+0000" 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.454619+0000" 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.499846+0000" 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 391, 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.467777+0000" 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.467792+0000" 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.469297+0000" 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.469972+0000" 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.472474+0000" 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.526744+0000" 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 394, 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.474246+0000" 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.474261+0000" 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.474945+0000" 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.475958+0000" 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.476941+0000" 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.539331+0000" 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 404, 2024-04-16T07:48:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.556204+0000" 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.556228+0000" 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.556475+0000" 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.556684+0000" 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.556848+0000" 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.581137+0000" 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 410, 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.608628+0000" 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.608652+0000" 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.608839+0000" 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.608968+0000" 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.609072+0000" 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.667260+0000" 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 411, 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.623085+0000" 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.623112+0000" 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.623475+0000" 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.623801+0000" 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.624120+0000" 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.690770+0000" 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 413, 2024-04-16T07:48:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.629133+0000" 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.629152+0000" 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.629373+0000" 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.633682+0000" 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.634264+0000" 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.721362+0000" 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 416, 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.646552+0000" 2024-04-16T07:48:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.646561+0000" 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.647003+0000" 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.647186+0000" 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.647416+0000" 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.727623+0000" 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 417, 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.650597+0000" 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.650607+0000" 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.658418+0000" 2024-04-16T07:48:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.658753+0000" 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.658932+0000" 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.739763+0000" 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 419, 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.675813+0000" 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.675828+0000" 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.679883+0000" 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.680769+0000" 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.681871+0000" 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.752103+0000" 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 436, 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.763062+0000" 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.763079+0000" 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.764501+0000" 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.764868+0000" 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.767228+0000" 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.828984+0000" 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 437, 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.774629+0000" 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.774653+0000" 2024-04-16T07:48:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.778963+0000" 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.780371+0000" 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.780526+0000" 2024-04-16T07:48:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.830403+0000" 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 440, 2024-04-16T07:48:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.811597+0000" 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.811615+0000" 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.811969+0000" 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.812212+0000" 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.812451+0000" 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.862597+0000" 2024-04-16T07:48:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 446, 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.840522+0000" 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.840537+0000" 2024-04-16T07:48:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.840798+0000" 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.840967+0000" 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.842409+0000" 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.888863+0000" 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 468, 2024-04-16T07:48:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.939126+0000" 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.939135+0000" 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.939799+0000" 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.941940+0000" 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.942667+0000" 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.998694+0000" 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 465, 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.934120+0000" 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.934135+0000" 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.934748+0000" 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.934957+0000" 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.935118+0000" 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.007946+0000" 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 469, 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.965164+0000" 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.965176+0000" 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.965813+0000" 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.965949+0000" 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.966196+0000" 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.032427+0000" 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 473, 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.000126+0000" 2024-04-16T07:48:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.000138+0000" 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.000627+0000" 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.000953+0000" 2024-04-16T07:48:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.001113+0000" 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.068903+0000" 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 476, 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.021058+0000" 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.021088+0000" 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.021298+0000" 2024-04-16T07:48:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.022142+0000" 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.022340+0000" 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.078485+0000" 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 479, 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.027980+0000" 2024-04-16T07:48:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.027995+0000" 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.028454+0000" 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.029412+0000" 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.029618+0000" 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.101112+0000" 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 480, 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.044240+0000" 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.044260+0000" 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.044592+0000" 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.045114+0000" 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.046088+0000" 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.105306+0000" 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 500, 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.180808+0000" 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.180827+0000" 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.180990+0000" 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.181092+0000" 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.181186+0000" 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.217537+0000" 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 501, 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.189838+0000" 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.189855+0000" 2024-04-16T07:48:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.190668+0000" 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.190990+0000" 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.191143+0000" 2024-04-16T07:48:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.230907+0000" 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 502, 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.193092+0000" 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.193103+0000" 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.193647+0000" 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.193953+0000" 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.195075+0000" 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.262259+0000" 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 30, 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.768918+0000" 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.768927+0000" 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.769082+0000" 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.859580+0000" 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.863202+0000" 2024-04-16T07:48:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.037431+0000" 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 145, 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.949541+0000" 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.949550+0000" 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.951560+0000" 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.957232+0000" 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:22.966099+0000" 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.240317+0000" 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 208, 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.224332+0000" 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.224346+0000" 2024-04-16T07:48:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.224551+0000" 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.225539+0000" 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.228003+0000" 2024-04-16T07:48:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.296893+0000" 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 216, 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.247231+0000" 2024-04-16T07:48:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.247246+0000" 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.247987+0000" 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.248727+0000" 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.249292+0000" 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.304402+0000" 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 226, 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.295449+0000" 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.295477+0000" 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.295757+0000" 2024-04-16T07:48:25.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.296355+0000" 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.297073+0000" 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.412 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.335661+0000" 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 259, 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.413 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.509861+0000" 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.509879+0000" 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.510079+0000" 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.510278+0000" 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.510677+0000" 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.415 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.531492+0000" 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 267, 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.569840+0000" 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.416 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.569865+0000" 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.570223+0000" 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.570414+0000" 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.417 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.570764+0000" 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.600226+0000" 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.418 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 294, 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.810445+0000" 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.810460+0000" 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.810626+0000" 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.419 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.810716+0000" 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.811194+0000" 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.420 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.841165+0000" 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 322, 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.018804+0000" 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.421 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.018813+0000" 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.019370+0000" 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.022981+0000" 2024-04-16T07:48:25.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.024985+0000" 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.070870+0000" 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.423 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 340, 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.119148+0000" 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.119160+0000" 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.119303+0000" 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.424 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.119394+0000" 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.119660+0000" 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.425 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.150117+0000" 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 348, 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.186419+0000" 2024-04-16T07:48:25.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.186428+0000" 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.186564+0000" 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.427 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.186655+0000" 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.186937+0000" 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.209432+0000" 2024-04-16T07:48:25.428 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 363, 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.288275+0000" 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.288289+0000" 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.429 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.288431+0000" 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.288541+0000" 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.288796+0000" 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.430 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.322905+0000" 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 378, 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.377349+0000" 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.431 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.377374+0000" 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.377652+0000" 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.377832+0000" 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.378639+0000" 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.414204+0000" 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 381, 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.433 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.393785+0000" 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.393799+0000" 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.395560+0000" 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.434 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.400490+0000" 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.401538+0000" 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.435 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.428306+0000" 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 386, 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.433707+0000" 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.433717+0000" 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.436 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.433887+0000" 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.434013+0000" 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.434318+0000" 2024-04-16T07:48:25.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.480411+0000" 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 400, 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.438 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.531740+0000" 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.531769+0000" 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.532549+0000" 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.532972+0000" 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.535098+0000" 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.440 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.563209+0000" 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 407, 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.589848+0000" 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.441 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.589874+0000" 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.590120+0000" 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.590291+0000" 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.442 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.590672+0000" 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.624424+0000" 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.443 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 431, 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.744338+0000" 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.744353+0000" 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.744969+0000" 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.745444+0000" 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.746599+0000" 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.802123+0000" 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.445 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 432, 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.756111+0000" 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.756121+0000" 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.757192+0000" 2024-04-16T07:48:25.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.757717+0000" 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.758751+0000" 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.447 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.821478+0000" 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 447, 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.845106+0000" 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.448 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.845119+0000" 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.845793+0000" 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.847180+0000" 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.449 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.850497+0000" 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.892417+0000" 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 482, 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.059952+0000" 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.059964+0000" 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.451 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.063464+0000" 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.063822+0000" 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.064442+0000" 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.452 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.111652+0000" 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 484, 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.069395+0000" 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.453 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.069405+0000" 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.069686+0000" 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.454 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.070927+0000" 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.073790+0000" 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.123427+0000" 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.455 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 507, 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.260388+0000" 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.260411+0000" 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.456 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.260793+0000" 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.261349+0000" 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.262023+0000" 2024-04-16T07:48:25.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "replicated_request", 2024-04-16T07:48:25.458 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 508, 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "RepRequest": { 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "reqid": "client.4221.0:7863", 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "pgid": "3.b", 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 41, 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "min_epoch": 40, 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "oid": "3:d5d813c5:::benchmark_data_smithi079_31310_object3199:head", 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "from": "3" 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "replicated_request", 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 509, 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.459 INFO:teuthology.orchestra.run.smithi079.stdout: "RepRequest": { 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: "reqid": "client.4221.0:7864", 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: "pgid": "3.b", 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 41, 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: "min_epoch": 40, 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: "oid": "3:d456eb02:::benchmark_data_smithi079_31310_object3200:head", 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: "from": "3" 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 46, 2024-04-16T07:48:25.460 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.821764+0000" 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.821783+0000" 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.822476+0000" 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.461 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.862324+0000" 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:21.864310+0000" 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.462 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.201113+0000" 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 204, 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.199575+0000" 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.463 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.199588+0000" 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.200075+0000" 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.200275+0000" 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.464 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.201280+0000" 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.291547+0000" 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 240, 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.357444+0000" 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.357468+0000" 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.359519+0000" 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.466 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.359703+0000" 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.360695+0000" 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.467 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.385250+0000" 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 244, 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.385991+0000" 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.468 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.386003+0000" 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.386160+0000" 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.469 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.386248+0000" 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.386511+0000" 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.419349+0000" 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 282, 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.710930+0000" 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.710952+0000" 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.471 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.711182+0000" 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.711311+0000" 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.711683+0000" 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.472 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.739853+0000" 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 287, 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.745730+0000" 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.473 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.745746+0000" 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.746952+0000" 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.747400+0000" 2024-04-16T07:48:25.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.747773+0000" 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.797179+0000" 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.475 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 289, 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.766124+0000" 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.766139+0000" 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.766387+0000" 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.766591+0000" 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.768850+0000" 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.830800+0000" 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 298, 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.839553+0000" 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.839569+0000" 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.839791+0000" 2024-04-16T07:48:25.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.839968+0000" 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.840534+0000" 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.479 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.480 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.481 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.481 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.890771+0000" 2024-04-16T07:48:25.481 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.481 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 302, 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.865429+0000" 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.865441+0000" 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.482 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.865614+0000" 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.865698+0000" 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.867160+0000" 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.483 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.899678+0000" 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.484 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 303, 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.868227+0000" 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.868237+0000" 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.868372+0000" 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.485 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.868653+0000" 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.868909+0000" 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:23.912997+0000" 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 327, 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.487 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.038685+0000" 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.038704+0000" 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.038905+0000" 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.488 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.041145+0000" 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.044000+0000" 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.489 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.077697+0000" 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 328, 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.046664+0000" 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.046680+0000" 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.047585+0000" 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.048087+0000" 2024-04-16T07:48:25.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.052540+0000" 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.102253+0000" 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.492 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 332, 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.078734+0000" 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.078753+0000" 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.078991+0000" 2024-04-16T07:48:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.079142+0000" 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.079471+0000" 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.494 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.117757+0000" 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 333, 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.080601+0000" 2024-04-16T07:48:25.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.080613+0000" 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.080907+0000" 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.081083+0000" 2024-04-16T07:48:25.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.087523+0000" 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.126301+0000" 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 336, 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.096124+0000" 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.096138+0000" 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.498 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.096361+0000" 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.096771+0000" 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.098433+0000" 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.499 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.149606+0000" 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 337, 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.097636+0000" 2024-04-16T07:48:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.097646+0000" 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.097967+0000" 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.501 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.098137+0000" 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.100441+0000" 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.155081+0000" 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 342, 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.134599+0000" 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.503 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.134614+0000" 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.135094+0000" 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.135203+0000" 2024-04-16T07:48:25.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.135448+0000" 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.184064+0000" 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.505 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 344, 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.156386+0000" 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.156410+0000" 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.156610+0000" 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.506 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.156697+0000" 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.156947+0000" 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.190617+0000" 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 356, 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.249234+0000" 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.508 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.249253+0000" 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.249560+0000" 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.509 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.249674+0000" 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.250201+0000" 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.286504+0000" 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.510 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 359, 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.255272+0000" 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.255291+0000" 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.511 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.255568+0000" 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.255733+0000" 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.260680+0000" 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.512 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.292298+0000" 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 365, 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.513 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.300653+0000" 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.300671+0000" 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.300815+0000" 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.514 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.301037+0000" 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.301565+0000" 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.346133+0000" 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 368, 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.315915+0000" 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.315938+0000" 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.317005+0000" 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.516 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.317164+0000" 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.317660+0000" 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.354983+0000" 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 382, 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.406231+0000" 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.518 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.406242+0000" 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.406400+0000" 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.406491+0000" 2024-04-16T07:48:25.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.406760+0000" 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.437981+0000" 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.520 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 383, 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.406889+0000" 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.406897+0000" 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.407050+0000" 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.521 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.407136+0000" 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.408135+0000" 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.443298+0000" 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 406, 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.582987+0000" 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.582999+0000" 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.583234+0000" 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.583366+0000" 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.583718+0000" 2024-04-16T07:48:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.605584+0000" 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 421, 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.525 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.692287+0000" 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.692302+0000" 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.692744+0000" 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.693061+0000" 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.693841+0000" 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.750763+0000" 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 422, 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.705901+0000" 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.705913+0000" 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.528 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.708587+0000" 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.709033+0000" 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.710381+0000" 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.529 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.797857+0000" 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 425, 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.530 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.710616+0000" 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.710626+0000" 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.713509+0000" 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.531 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.713755+0000" 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.720532+0000" 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.802867+0000" 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.532 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 442, 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.831055+0000" 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.831066+0000" 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.831259+0000" 2024-04-16T07:48:25.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.831476+0000" 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.832450+0000" 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.534 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.878581+0000" 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 444, 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.535 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.835669+0000" 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.835681+0000" 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.835894+0000" 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.536 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.836111+0000" 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.840281+0000" 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.902867+0000" 2024-04-16T07:48:25.537 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 456, 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.887015+0000" 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.887024+0000" 2024-04-16T07:48:25.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.887162+0000" 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.887401+0000" 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.539 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.888209+0000" 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.930331+0000" 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.540 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 457, 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.889389+0000" 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.889399+0000" 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.889610+0000" 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.541 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.889761+0000" 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.890082+0000" 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.955955+0000" 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 460, 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.901261+0000" 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.543 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.901280+0000" 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.903130+0000" 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.903290+0000" 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.544 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.907276+0000" 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:24.997130+0000" 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 497, 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:48:25.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.158502+0000" 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.158518+0000" 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.158696+0000" 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:48:25.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.158861+0000" 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.159837+0000" 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:48:25.192491+0000" 2024-04-16T07:48:25.547 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.548 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.548 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:48:25.548 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-16T07:48:25.548 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-16T07:48:25.548 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:48:25.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:25.758+0000 7fbd60406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:25.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:25.766+0000 7fbd60406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:28.488 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:48:28.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:28.645+0000 7f4d272ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:28.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:28.654+0000 7f4d272ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:30.550 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 1, 3, 0] 2024-04-16T07:48:30.551 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:48:30.551 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [2, 1, 3, 0] 2024-04-16T07:48:30.551 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T07:48:30.551 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:48:30.637 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T07:48:31.206 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:48:31.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:31.327+0000 7f7c90d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:31.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:31.333+0000 7f7c90d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:35.639 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [2, 3, 0] 2024-04-16T07:48:35.639 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:48:35.639 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-16T07:48:35.639 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:48:35.639 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:48:35.643 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:48:35.643 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:48:35.757 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:35.761 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:48:40.763 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:48:40.869 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:48:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:40.872 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:48:45.755 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-16T07:48:45.873 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:48:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:48:45.902 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:48:45.981 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-16T07:48:45.981 INFO:teuthology.orchestra.run.smithi079.stdout: "ops_in_flight": [] 2024-04-16T07:48:45.981 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-16T07:48:47.903 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:48:48.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:48.053+0000 7fbd7406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:48.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:48.060+0000 7fbd7406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:49.471 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Removed 4661 objects 2024-04-16T07:48:49.471 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Clean up completed and total clean up time :97.1597 2024-04-16T07:48:50.350 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-16T07:48:50.351 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-16T07:48:50.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:50.467+0000 7f40e786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:50.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:50.480+0000 7f40e786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:50.518 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:48:50.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:50.629+0000 7fd32ec64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:50.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:50.635+0000 7fd32ec64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:50.766 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'unique_pool_0' removed 2024-04-16T07:48:50.784 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-16T07:48:50.785 INFO:tasks.radosbench:Beginning radosbench... 2024-04-16T07:48:50.785 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_1 2024-04-16T07:48:50.786 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_1 16 2024-04-16T07:48:50.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:50.974+0000 7f8297c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:50.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:50.988+0000 7f8296a0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:50.991 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-04-16T07:48:50.991 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:48:50.991 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:48:51.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:51.100+0000 7f9cd902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:51.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:51.108+0000 7f9cd882c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:51.298 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:48:51.299 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:48:51.311 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-04-16T07:48:51.082048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":23,"acting":23,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":148456,"kb_used_data":5064,"kb_used_omap":0,"kb_used_meta":143360,"kb_avail":377338904,"statfs":{"total":386547056640,"available":386395037696,"internally_reserved":0,"allocated":5185536,"data_stored":59454884,"data_compressed":2803562,"data_compressed_allocated":3694592,"data_compressed_original":58867712,"omap_allocated":0,"internal_metadata":146800640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23265304,"num_objects":-356,"num_object_clones":0,"num_object_copies":-1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-8725,"num_write_kb":-283521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-39,"num_bytes_recovered":-2228224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.329855"},"pg_stats":[{"pgid":"1.0","version":"12'97","reported_seq":231,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-16T07:48:47.787279+0000","last_change":"2024-04-16T07:48:22.913279+0000","last_active":"2024-04-16T07:48:47.787279+0000","last_peered":"2024-04-16T07:48:47.787279+0000","last_clean":"2024-04-16T07:48:47.787279+0000","last_became_active":"2024-04-16T07:48:22.900434+0000","last_became_peered":"2024-04-16T07:48:22.900434+0000","last_unstale":"2024-04-16T07:48:47.787279+0000","last_undegraded":"2024-04-16T07:48:47.787279+0000","last_fullsized":"2024-04-16T07:48:47.787279+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":92,"reported_epoch":42,"state":"peering","last_fresh":"2024-04-16T07:48:22.816193+0000","last_change":"2024-04-16T07:48:20.806046+0000","last_active":"2024-04-16T07:48:20.806046+0000","last_peered":"2024-04-16T07:48:06.297907+0000","last_clean":"2024-04-16T07:48:06.297907+0000","last_became_active":"2024-04-16T07:48:06.286207+0000","last_became_peered":"2024-04-16T07:48:06.286207+0000","last_unstale":"2024-04-16T07:48:22.816193+0000","last_undegraded":"2024-04-16T07:48:22.816193+0000","last_fullsized":"2024-04-16T07:48:22.816193+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":79,"reported_epoch":42,"state":"peering","last_fresh":"2024-04-16T07:48:23.009064+0000","last_change":"2024-04-16T07:48:45.796584+0000","last_active":"2024-04-16T07:48:45.796584+0000","last_peered":"2024-04-16T07:48:23.009064+0000","last_clean":"2024-04-16T07:48:23.009064+0000","last_became_active":"2024-04-16T07:48:23.004812+0000","last_became_peered":"2024-04-16T07:48:23.004812+0000","last_unstale":"2024-04-16T07:48:23.009064+0000","last_undegraded":"2024-04-16T07:48:23.009064+0000","last_fullsized":"2024-04-16T07:48:23.009064+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":74,"reported_epoch":41,"state":"stale+active+clean","last_fresh":"2024-04-16T07:48:22.812696+0000","last_change":"2024-04-16T07:47:55.842517+0000","last_active":"2024-04-16T07:48:22.812696+0000","last_peered":"2024-04-16T07:48:22.812696+0000","last_clean":"2024-04-16T07:48:22.812696+0000","last_became_active":"2024-04-16T07:47:55.833970+0000","last_became_peered":"2024-04-16T07:47:55.833970+0000","last_unstale":"2024-04-16T07:48:45.752233+0000","last_undegraded":"2024-04-16T07:48:22.812696+0000","last_fullsized":"2024-04-16T07:48:22.812696+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":42,"state":"peering","last_fresh":"2024-04-16T07:48:23.031536+0000","last_change":"2024-04-16T07:48:45.814911+0000","last_active":"2024-04-16T07:48:45.814911+0000","last_peered":"2024-04-16T07:48:23.031536+0000","last_clean":"2024-04-16T07:48:23.031536+0000","last_became_active":"2024-04-16T07:48:23.020758+0000","last_became_peered":"2024-04-16T07:48:23.020758+0000","last_unstale":"2024-04-16T07:48:23.031536+0000","last_undegraded":"2024-04-16T07:48:23.031536+0000","last_fullsized":"2024-04-16T07:48:23.031536+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-16T07:48:47.809672+0000","last_change":"2024-04-16T07:48:22.907961+0000","last_active":"2024-04-16T07:48:47.809672+0000","last_peered":"2024-04-16T07:48:47.809672+0000","last_clean":"2024-04-16T07:48:47.809672+0000","last_became_active":"2024-04-16T07:48:22.897330+0000","last_became_peered":"2024-04-16T07:48:22.897330+0000","last_unstale":"2024-04-16T07:48:47.809672+0000","last_undegraded":"2024-04-16T07:48:47.809672+0000","last_fullsized":"2024-04-16T07:48:47.809672+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-16T07:48:47.802461+0000","last_change":"2024-04-16T07:48:22.899313+0000","last_active":"2024-04-16T07:48:47.802461+0000","last_peered":"2024-04-16T07:48:47.802461+0000","last_clean":"2024-04-16T07:48:47.802461+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:48:22.892718+0000","last_unstale":"2024-04-16T07:48:47.802461+0000","last_undegraded":"2024-04-16T07:48:47.802461+0000","last_fullsized":"2024-04-16T07:48:47.802461+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-16T07:48:47.819091+0000","last_change":"2024-04-16T07:48:22.941186+0000","last_active":"2024-04-16T07:48:47.819091+0000","last_peered":"2024-04-16T07:48:47.819091+0000","last_clean":"2024-04-16T07:48:47.819091+0000","last_became_active":"2024-04-16T07:48:22.927045+0000","last_became_peered":"2024-04-16T07:48:22.927045+0000","last_unstale":"2024-04-16T07:48:47.819091+0000","last_undegraded":"2024-04-16T07:48:47.819091+0000","last_fullsized":"2024-04-16T07:48:47.819091+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":92,"reported_epoch":42,"state":"peering","last_fresh":"2024-04-16T07:48:22.786949+0000","last_change":"2024-04-16T07:48:20.813282+0000","last_active":"2024-04-16T07:48:20.813282+0000","last_peered":"2024-04-16T07:48:06.281829+0000","last_clean":"2024-04-16T07:48:06.281829+0000","last_became_active":"2024-04-16T07:48:06.275466+0000","last_became_peered":"2024-04-16T07:48:06.275466+0000","last_unstale":"2024-04-16T07:48:22.786949+0000","last_undegraded":"2024-04-16T07:48:22.786949+0000","last_fullsized":"2024-04-16T07:48:22.786949+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":20,"acting":20,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":34,"seq":146028888118,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28512,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94343328,"statfs":{"total":96636764160,"available":96607567872,"internally_reserved":0,"allocated":942080,"data_stored":8920934,"data_compressed":418966,"data_compressed_allocated":552960,"data_compressed_original":8765440,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40608,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94331232,"statfs":{"total":96636764160,"available":96595181568,"internally_reserved":0,"allocated":811008,"data_stored":6889294,"data_compressed":322494,"data_compressed_allocated":425984,"data_compressed_original":6733824,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084367,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":42260,"kb_used_data":2188,"kb_used_omap":0,"kb_used_meta":40064,"kb_avail":94329580,"statfs":{"total":96636764160,"available":96593489920,"internally_reserved":0,"allocated":2240512,"data_stored":30471007,"data_compressed":1440856,"data_compressed_allocated":1896448,"data_compressed_original":30343168,"omap_allocated":0,"internal_metadata":41025536},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37076,"kb_used_data":1164,"kb_used_omap":0,"kb_used_meta":35904,"kb_avail":94334764,"statfs":{"total":96636764160,"available":96598798336,"internally_reserved":0,"allocated":1191936,"data_stored":13173649,"data_compressed":621246,"data_compressed_allocated":819200,"data_compressed_original":13025280,"omap_allocated":0,"internal_metadata":36765696},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:48:51.770 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'unique_pool_1' created 2024-04-16T07:48:51.791 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-04-16T07:48:51.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:51.928+0000 7f271be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:51.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:51.941+0000 7f271be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:52.752 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:48:52.778 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-04-16T07:48:52.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:52.901+0000 7fd0f6f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:52.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:52.912+0000 7fd0f6f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:53.800 DEBUG:teuthology.orchestra.run.smithi079:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-04-16T07:48:53.803 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-16T07:48:53.803 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-16T07:48:53.869 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:hints = 1 2024-04-16T07:48:53.869 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-16T07:48:53.869 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_35014 2024-04-16T07:48:53.869 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:48:53.870 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-04-16T07:48:54.869 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1 16 197 181 11.3118 11.3125 0.0934284 0.0831123 2024-04-16T07:48:55.143 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:48:55.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:55.286+0000 7ff0142ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:55.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:55.294+0000 7ff0142ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:55.870 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2 16 393 377 11.7803 12.25 0.053013 0.0834773 2024-04-16T07:48:56.312 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-04-16T07:48:56.312 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:48:56.312 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-16T07:48:56.313 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-16T07:48:56.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:56.466+0000 7f228906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:56.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:56.479+0000 7f228906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:48:56.555 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:48:56.556 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:48:56.556 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:48:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:56.560 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:48:56.561 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-16T07:48:56.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:56.733+0000 7fd28a26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:56.745+0000 7fd28a26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:56.870 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3 16 574 558 11.624 11.3125 0.0343498 0.084738 2024-04-16T07:48:56.878 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:48:56.879 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:48:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:56.884 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:48:56.884 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-16T07:48:57.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:57.039+0000 7f4a2b464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:57.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:57.052+0000 7f4a2b464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:48:57.075 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:48:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:57.078 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:48:57.078 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-16T07:48:57.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:57.231+0000 7f5c1d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:57.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:57.245+0000 7f5c1d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:57.279 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-16T07:48:57.279 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-16T07:48:57.279 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-16T07:48:57.280 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-16T07:48:57.280 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-16T07:48:57.280 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-16T07:48:57.280 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-16T07:48:57.280 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-16T07:48:57.280 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-16T07:48:57.280 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-16T07:48:57.280 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-16T07:48:57.280 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-16T07:48:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:48:57.285 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-16T07:48:57.665 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:48:57.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:57.817+0000 7fc88846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:57.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:48:57.827+0000 7fc88846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:48:57.870 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 4 16 759 743 11.6084 11.5625 0.117286 0.0846896 2024-04-16T07:48:58.870 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 5 16 948 932 11.649 11.8125 0.0937878 0.0852152 2024-04-16T07:48:59.870 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 6 16 1125 1109 11.5511 11.0625 0.110617 0.0856825 2024-04-16T07:49:00.160 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:49:00.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:00.299+0000 7f3db3eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:00.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:00.306+0000 7f3db3eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:00.870 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 7 16 1313 1297 11.5793 11.75 0.0377624 0.0857791 2024-04-16T07:49:01.870 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 8 16 1492 1476 11.5302 11.1875 0.0675703 0.0861161 2024-04-16T07:49:02.286 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-04-16T07:49:02.286 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:49:02.286 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [2, 3, 0, 1] 2024-04-16T07:49:02.287 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T07:49:02.287 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:49:02.373 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T07:49:02.647 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:49:02.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:02.759+0000 7f258891d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:02.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:02.764+0000 7f258891d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:02.870 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 9 16 1576 1560 10.8323 5.25 0.016369 0.0863397 2024-04-16T07:49:03.871 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 10 16 1576 1560 9.74898 0 - 0.0863397 2024-04-16T07:49:04.871 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 11 16 1576 1560 8.86268 0 - 0.0863397 2024-04-16T07:49:05.871 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 12 16 1576 1560 8.12409 0 - 0.0863397 2024-04-16T07:49:06.871 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 13 16 1576 1560 7.49912 0 - 0.0863397 2024-04-16T07:49:07.374 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [2, 3, 0] 2024-04-16T07:49:07.374 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:49:07.374 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [2, 3, 0] 2024-04-16T07:49:07.374 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T07:49:07.375 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T07:49:07.460 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T07:49:07.871 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 14 16 1576 1560 6.96345 0 - 0.0863397 2024-04-16T07:49:08.871 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 15 16 1576 1560 6.49921 0 - 0.0863397 2024-04-16T07:49:09.872 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 16 16 1576 1560 6.09299 0 - 0.0863397 2024-04-16T07:49:10.872 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 17 16 1576 1560 5.73458 0 - 0.0863397 2024-04-16T07:49:11.872 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 18 16 1576 1560 5.41598 0 - 0.0863397 2024-04-16T07:49:12.461 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1, 3] live_osds: [2, 0] 2024-04-16T07:49:12.461 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:49:12.461 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-16T07:49:12.461 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-16T07:49:12.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-16T07:49:12.465 INFO:tasks.ceph.osd.3:Started 2024-04-16T07:49:12.465 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:12.580 INFO:teuthology.orchestra.run.smithi167.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:12.584 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:12.872 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 19 16 1576 1560 5.13092 0 - 0.0863397 2024-04-16T07:49:13.872 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:49:13.873762+0000 min lat: 0.0163555 max lat: 0.209377 avg lat: 0.0863397 2024-04-16T07:49:13.872 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:49:13.872 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 20 16 1576 1560 4.87437 0 - 0.0863397 2024-04-16T07:49:14.872 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 21 16 1576 1560 4.64225 0 - 0.0863397 2024-04-16T07:49:15.872 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 22 16 1576 1560 4.43124 0 - 0.0863397 2024-04-16T07:49:16.873 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 23 16 1576 1560 4.23858 0 - 0.0863397 2024-04-16T07:49:17.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:17.689 INFO:teuthology.orchestra.run.smithi167.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:17.693 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-16T07:49:17.873 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 24 16 1576 1560 4.06197 0 - 0.0863397 2024-04-16T07:49:18.873 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 25 16 1576 1560 3.89949 0 - 0.0863397 2024-04-16T07:49:19.873 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 26 16 1576 1560 3.7495 0 - 0.0863397 2024-04-16T07:49:20.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 27 16 1576 1560 3.61063 0 - 0.0863397 2024-04-16T07:49:21.873 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 28 16 1576 1560 3.48169 0 - 0.0863397 2024-04-16T07:49:22.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-16T07:49:22.840 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-16T07:49:22.840 INFO:teuthology.orchestra.run.smithi167.stdout: "ops_in_flight": [ 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "peering_event", 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 86, 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "PeeringEvent": { 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "from": "2", 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "pgid": "2.6", 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "sent": 51, 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "requested": 50, 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "evt": "epoch_sent: 51 epoch_requested: 50 MLogRec from 2 log log((0'0,0'0], crt=0'0) pi ([0,0] all_participants={} intervals=[]) pg_lease(ru 0.000000000s ub 180.123367310s int 16.000000000s) +create_info", 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:49:22.828676+0000" 2024-04-16T07:49:22.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:49:22.828686+0000" 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:49:22.829166+0000" 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:22.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:49:22.831385+0000" 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:49:22.834457+0000" 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:49:22.843 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:49:22.834958+0000" 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:49:22.835027+0000" 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:49:22.844 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "peering_event", 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "id": 16777269, 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "detail": { 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "PeeringEvent": { 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "from": "3", 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "pgid": "2.6", 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "sent": 51, 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "requested": 51, 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "evt": "epoch_sent: 51 epoch_requested: 51 ActivateCommitted from 51 processed at 51", 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-16T07:49:22.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:49:22.839870+0000" 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-16T07:49:22.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-16T07:49:22.839940+0000" 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-16T07:49:22.847 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-16T07:49:22.873 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 29 16 1576 1560 3.36163 0 - 0.0863397 2024-04-16T07:49:23.873 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 30 16 1576 1560 3.24958 0 - 0.0863397 2024-04-16T07:49:24.873 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 31 16 1576 1560 3.14476 0 - 0.0863397 2024-04-16T07:49:25.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 32 16 1576 1560 3.04648 0 - 0.0863397 2024-04-16T07:49:26.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 33 16 1576 1560 2.95416 0 - 0.0863397 2024-04-16T07:49:27.853 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [2, 0, 3] 2024-04-16T07:49:27.853 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:49:27.853 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-16T07:49:27.853 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:49:27.854 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:49:27.857 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:49:27.857 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:49:27.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 34 16 1576 1560 2.86728 0 - 0.0863397 2024-04-16T07:49:27.976 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:27.980 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:49:28.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 35 16 1576 1560 2.78536 0 - 0.0863397 2024-04-16T07:49:29.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 36 16 1576 1560 2.70799 0 - 0.0863397 2024-04-16T07:49:30.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 37 16 1576 1560 2.6348 0 - 0.0863397 2024-04-16T07:49:31.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 38 16 1576 1560 2.56546 0 - 0.0863397 2024-04-16T07:49:32.145 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-16T07:49:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:49:32.299 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:49:32.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 39 16 1576 1560 2.49968 0 - 0.0863397 2024-04-16T07:49:32.980 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:49:33.084 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-16T07:49:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T07:49:33.087 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-16T07:49:33.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:49:33.876229+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.378171 2024-04-16T07:49:33.875 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:49:33.875 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 40 16 1707 1691 2.64185 0.264113 0.0772417 0.378171 2024-04-16T07:49:34.300 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:49:34.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:34.450+0000 7f25564ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:34.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:34.457+0000 7f25564ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:34.509 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-16T07:49:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-16T07:49:34.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:49:34.874 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 41 16 1915 1899 2.89446 13 0.135606 0.344511 2024-04-16T07:49:35.875 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 42 16 1920 1904 2.83298 0.3125 0.0182366 0.343829 2024-04-16T07:49:36.761 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:49:36.875 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 43 16 1934 1918 2.78745 0.875 1.30165 0.348198 2024-04-16T07:49:36.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:36.875+0000 7f2e7baad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:36.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:36.881+0000 7f2e7baad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:37.875 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 44 16 1934 1918 2.7241 0 - 0.348198 2024-04-16T07:49:38.088 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-16T07:49:38.197 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: "ops_in_flight": [ 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 147, 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.110319+0000" 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.198 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.110330+0000" 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.111442+0000" 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.112289+0000" 2024-04-16T07:49:38.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.115272+0000" 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.200 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 149, 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.120994+0000" 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.121011+0000" 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.201 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.121179+0000" 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.121334+0000" 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.202 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.121513+0000" 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.203 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "background_recovery", 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 152, 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: "pgid": "4.d", 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_detail": {} 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 54, 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.130223+0000" 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.130238+0000" 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.130956+0000" 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.205 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.136592+0000" 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.138048+0000" 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.206 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.207 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.207 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.207 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.104899+0000" 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 16777338, 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:38.027824+0000" 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.208 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:38.027849+0000" 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:38.028001+0000" 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.209 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:38.028122+0000" 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:38.028342+0000" 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:38.031101+0000" 2024-04-16T07:49:38.210 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 43, 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.077310+0000" 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.077335+0000" 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.211 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.078663+0000" 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.080538+0000" 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.212 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.081504+0000" 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.000611+0000" 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.213 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 45, 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.082727+0000" 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.082737+0000" 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.214 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.083169+0000" 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.083381+0000" 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.084762+0000" 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.215 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.091193+0000" 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 141, 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.216 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.072514+0000" 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.072532+0000" 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.074184+0000" 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.217 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.074330+0000" 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.079760+0000" 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.218 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.126510+0000" 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 62, 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.219 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.168766+0000" 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.168775+0000" 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.168942+0000" 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.220 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.169083+0000" 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.169845+0000" 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.221 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.168283+0000" 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 49, 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.093614+0000" 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.093639+0000" 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.102369+0000" 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.104338+0000" 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.105304+0000" 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.224 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 56, 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.151584+0000" 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.151610+0000" 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.154889+0000" 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.225 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.155068+0000" 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.158646+0000" 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.226 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 59, 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.227 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.164144+0000" 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.164162+0000" 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.164387+0000" 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.165151+0000" 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.171000+0000" 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.229 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 61, 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.167637+0000" 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.230 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.167649+0000" 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.167799+0000" 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.167926+0000" 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.171151+0000" 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.232 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 65, 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.173354+0000" 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.173365+0000" 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.233 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.173531+0000" 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.173961+0000" 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.234 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.177868+0000" 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 125, 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.005406+0000" 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.005423+0000" 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.236 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.005719+0000" 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.005874+0000" 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.237 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.010630+0000" 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 134, 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.041581+0000" 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.041606+0000" 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.239 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.041798+0000" 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.043859+0000" 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.046891+0000" 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.240 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 153, 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.159106+0000" 2024-04-16T07:49:38.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.159123+0000" 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.159418+0000" 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.159592+0000" 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.162003+0000" 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 158, 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.172640+0000" 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.172658+0000" 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.172804+0000" 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.172945+0000" 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.176398+0000" 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.245 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 164, 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.246 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.198584+0000" 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.198602+0000" 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.200624+0000" 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.247 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.200822+0000" 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.204417+0000" 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.248 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "background_recovery", 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 33554530, 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: "pgid": "4.5", 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_detail": { 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: "oid": "4:a8feccd5:::benchmark_data_smithi079_35014_object1909:head", 2024-04-16T07:49:38.249 INFO:teuthology.orchestra.run.smithi079.stdout: "version": "0'0" 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "background_recovery", 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 33554532, 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: "pgid": "4.1", 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_detail": { 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: "oid": "4:87ac69b3:::benchmark_data_smithi079_35014_object1898:head", 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: "version": "0'0" 2024-04-16T07:49:38.250 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "background_recovery", 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 33554564, 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: "pgid": "4.1", 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_detail": {} 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 52, 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.110375+0000" 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.110392+0000" 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.110752+0000" 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.252 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.113158+0000" 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.115445+0000" 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.253 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.055826+0000" 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "client_request", 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 64, 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "detail": { 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.254 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.169916+0000" 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.169923+0000" 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.170204+0000" 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.255 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.170609+0000" 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:35.172048+0000" 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "blocking_event": { 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "0.000000" 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time_event": { 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-04-16T07:49:36.184267+0000" 2024-04-16T07:49:38.256 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-16T07:49:38.257 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-16T07:49:38.257 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-16T07:49:38.875 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 45 16 1934 1918 2.66356 0 - 0.348198 2024-04-16T07:49:39.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-16T07:49:39.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-16T07:49:39.875 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 46 16 1934 1918 2.60566 0 - 0.348198 2024-04-16T07:49:40.875 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 47 16 1934 1918 2.55022 0 - 0.348198 2024-04-16T07:49:41.875 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 48 16 1934 1918 2.49709 0 - 0.348198 2024-04-16T07:49:42.876 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 49 16 1934 1918 2.44613 0 - 0.348198 2024-04-16T07:49:43.258 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 0, 3, 1] 2024-04-16T07:49:43.258 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-16T07:49:43.259 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:43.380+0000 7fa36a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:43.387+0000 7fa36a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:43.572 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:49:43.572 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:49:43.584 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-04-16T07:49:43.093360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71893536,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1225,"num_write_kb":71526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1906,"ondisk_log_size":1906,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145608,"kb_used_data":21864,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341752,"statfs":{"total":386547056640,"available":386397954048,"internally_reserved":0,"allocated":22388736,"data_stored":329411380,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28508160,"num_objects":-435,"num_object_clones":0,"num_object_copies":-1305,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-435,"num_write_kb":-27840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.340066"},"pg_stats":[{"pgid":"4.f","version":"55'115","reported_seq":150,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.020003+0000","last_change":"2024-04-16T07:49:22.892700+0000","last_active":"2024-04-16T07:49:36.020003+0000","last_peered":"2024-04-16T07:49:36.020003+0000","last_clean":"2024-04-16T07:49:36.020003+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:36.020003+0000","last_undegraded":"2024-04-16T07:49:36.020003+0000","last_fullsized":"2024-04-16T07:49:36.020003+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":193,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.540096+0000","last_change":"2024-04-16T07:49:36.541161+0000","last_active":"2024-04-16T07:49:36.540096+0000","last_peered":"2024-04-16T07:49:36.540096+0000","last_clean":"2024-04-16T07:49:02.265533+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:36.540096+0000","last_undegraded":"2024-04-16T07:49:36.540096+0000","last_fullsized":"2024-04-16T07:49:36.540096+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'101","reported_seq":109,"reported_epoch":55,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:36.137708+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:36.137708+0000","last_peered":"2024-04-16T07:49:36.137708+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:36.137708+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:36.137708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:49:21.822703+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:49:22.788948+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":152,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.495621+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:36.495621+0000","last_peered":"2024-04-16T07:49:36.495621+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:36.495621+0000","last_undegraded":"2024-04-16T07:49:36.495621+0000","last_fullsized":"2024-04-16T07:49:36.495621+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'126","reported_seq":126,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.170181+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:36.170181+0000","last_peered":"2024-04-16T07:49:36.170181+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:36.170181+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:36.170181+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"53'110","reported_seq":107,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:35.979748+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:35.979748+0000","last_peered":"2024-04-16T07:49:35.979748+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:35.979748+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:35.979748+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":124,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:35.984828+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:35.984828+0000","last_peered":"2024-04-16T07:49:35.984828+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:35.984828+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:35.984828+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":231,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.852687+0000","last_change":"2024-04-16T07:49:22.853950+0000","last_active":"2024-04-16T07:49:22.852687+0000","last_peered":"2024-04-16T07:49:22.852687+0000","last_clean":"2024-04-16T07:48:22.913279+0000","last_became_active":"2024-04-16T07:49:22.847179+0000","last_became_peered":"2024-04-16T07:49:22.847179+0000","last_unstale":"2024-04-16T07:49:22.852687+0000","last_undegraded":"2024-04-16T07:49:22.852687+0000","last_fullsized":"2024-04-16T07:49:22.852687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.971860+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:35.971860+0000","last_peered":"2024-04-16T07:49:35.971860+0000","last_clean":"2024-04-16T07:48:22.915206+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:35.971860+0000","last_undegraded":"2024-04-16T07:49:35.971860+0000","last_fullsized":"2024-04-16T07:49:35.971860+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"48'105","reported_seq":127,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.847019+0000","last_change":"2024-04-16T07:49:22.848219+0000","last_active":"2024-04-16T07:49:22.847019+0000","last_peered":"2024-04-16T07:49:22.847019+0000","last_clean":"2024-04-16T07:49:02.322013+0000","last_became_active":"2024-04-16T07:49:22.843784+0000","last_became_peered":"2024-04-16T07:49:22.843784+0000","last_unstale":"2024-04-16T07:49:22.847019+0000","last_undegraded":"2024-04-16T07:49:22.847019+0000","last_fullsized":"2024-04-16T07:49:22.847019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:49:22.790566+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":154,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.772504+0000","last_change":"2024-04-16T07:49:22.862015+0000","last_active":"2024-04-16T07:49:35.772504+0000","last_peered":"2024-04-16T07:49:35.772504+0000","last_clean":"2024-04-16T07:49:35.772504+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:35.772504+0000","last_undegraded":"2024-04-16T07:49:35.772504+0000","last_fullsized":"2024-04-16T07:49:35.772504+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.897013+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:35.897013+0000","last_peered":"2024-04-16T07:49:35.897013+0000","last_clean":"2024-04-16T07:47:55.842517+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:35.897013+0000","last_undegraded":"2024-04-16T07:49:35.897013+0000","last_fullsized":"2024-04-16T07:49:35.897013+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'119","reported_seq":156,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.078781+0000","last_change":"2024-04-16T07:49:36.046350+0000","last_active":"2024-04-16T07:49:36.078781+0000","last_peered":"2024-04-16T07:49:36.078781+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:36.078781+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:36.078781+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":107},{"shards":"2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.801564+0000","last_change":"2024-04-16T07:48:47.921810+0000","last_active":"2024-04-16T07:49:22.801564+0000","last_peered":"2024-04-16T07:49:22.801564+0000","last_clean":"2024-04-16T07:49:22.801564+0000","last_became_active":"2024-04-16T07:48:47.913008+0000","last_became_peered":"2024-04-16T07:48:47.913008+0000","last_unstale":"2024-04-16T07:49:22.801564+0000","last_undegraded":"2024-04-16T07:49:22.801564+0000","last_fullsized":"2024-04-16T07:49:22.801564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":134,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.151363+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:36.151363+0000","last_peered":"2024-04-16T07:49:36.151363+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:36.151363+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:36.151363+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.770984+0000","last_change":"2024-04-16T07:49:22.857045+0000","last_active":"2024-04-16T07:49:35.770984+0000","last_peered":"2024-04-16T07:49:35.770984+0000","last_clean":"2024-04-16T07:49:35.770984+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:35.770984+0000","last_undegraded":"2024-04-16T07:49:35.770984+0000","last_fullsized":"2024-04-16T07:49:35.770984+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:49:22.787540+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.864951+0000","last_change":"2024-04-16T07:49:22.866332+0000","last_active":"2024-04-16T07:49:22.864951+0000","last_peered":"2024-04-16T07:49:22.864951+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:49:22.858691+0000","last_became_peered":"2024-04-16T07:49:22.858691+0000","last_unstale":"2024-04-16T07:49:22.864951+0000","last_undegraded":"2024-04-16T07:49:22.864951+0000","last_fullsized":"2024-04-16T07:49:22.864951+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:49:21.774567+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.794837+0000","last_change":"2024-04-16T07:49:22.896203+0000","last_active":"2024-04-16T07:49:35.794837+0000","last_peered":"2024-04-16T07:49:35.794837+0000","last_clean":"2024-04-16T07:49:35.794837+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:35.794837+0000","last_undegraded":"2024-04-16T07:49:35.794837+0000","last_fullsized":"2024-04-16T07:49:35.794837+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":122,"reported_epoch":55,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:36.034266+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:36.034266+0000","last_peered":"2024-04-16T07:49:36.034266+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:36.034266+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:36.034266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.960858+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:35.960858+0000","last_peered":"2024-04-16T07:49:35.960858+0000","last_clean":"2024-04-16T07:48:22.871937+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:35.960858+0000","last_undegraded":"2024-04-16T07:49:35.960858+0000","last_fullsized":"2024-04-16T07:49:35.960858+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":71303168,"num_objects":1088,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1088,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1809,"ondisk_log_size":1809,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32624,"kb_used_data":5288,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":5414912,"data_stored":79083184,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928233988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38240,"kb_used_data":5784,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333600,"statfs":{"total":96636764160,"available":96597606400,"internally_reserved":0,"allocated":5922816,"data_stored":87602320,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:43.586 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 4.5 4.4 4.1 2024-04-16T07:49:43.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:43.694+0000 7fc6d60ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:43.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:43.704+0000 7fc6d60ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:43.876 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 50 16 1934 1918 2.39721 0 - 0.348198 2024-04-16T07:49:43.889 INFO:teuthology.orchestra.run.smithi079.stderr:instructing pg(s) [4.1,4.5] on osd.1 to force-recovery; instructing pg(s) [4.4] on osd.2 to force-recovery; 2024-04-16T07:49:43.889 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-16T07:49:44.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-04-16T07:49:44.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~5s 2024-04-16T07:49:44.876 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 51 16 1934 1918 2.3502 0 - 0.348198 2024-04-16T07:49:45.876 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 52 16 1934 1918 2.30501 0 - 0.348198 2024-04-16T07:49:46.876 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 53 16 1934 1918 2.26151 0 - 0.348198 2024-04-16T07:49:47.876 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 54 16 1934 1918 2.21963 0 - 0.348198 2024-04-16T07:49:48.876 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 55 16 1934 1918 2.17927 0 - 0.348198 2024-04-16T07:49:48.906 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 0, 3, 1] 2024-04-16T07:49:48.906 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-16T07:49:49.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:49.027+0000 7f34cae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:49.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:49.037+0000 7f34cae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:49.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 56 16 1934 1918 2.14036 0 - 0.348198 2024-04-16T07:49:50.158 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.0 to 1 (10000) 2024-04-16T07:49:50.175 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-16T07:49:50.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2024-04-16T07:49:50.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~11s 2024-04-16T07:49:50.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:50.289+0000 7f1ee146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:50.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:50.299+0000 7f1ee146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:50.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 57 16 1934 1918 2.10281 0 - 0.348198 2024-04-16T07:49:51.164 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.1 to 1 (10000) 2024-04-16T07:49:51.179 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-16T07:49:51.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:51.289+0000 7ff99d8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:51.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:51.298+0000 7ff99d8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:51.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 58 16 1934 1918 2.06655 0 - 0.348198 2024-04-16T07:49:52.164 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.2 to 1 (10000) 2024-04-16T07:49:52.181 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-16T07:49:52.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:52.292+0000 7f91f599d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:52.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:52.301+0000 7f91f599d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:52.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 59 16 1934 1918 2.03153 0 - 0.348198 2024-04-16T07:49:53.172 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.3 to 1 (10000) 2024-04-16T07:49:53.187 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-16T07:49:53.187 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:53.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:53.297+0000 7f87c039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:53.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:53.305+0000 7f87c039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:49:53.491 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:49:53.504 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-04-16T07:49:53.173824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71893536,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1225,"num_write_kb":71526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1906,"ondisk_log_size":1906,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145608,"kb_used_data":21864,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341752,"statfs":{"total":386547056640,"available":386397954048,"internally_reserved":0,"allocated":22388736,"data_stored":329411380,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.078782"},"pg_stats":[{"pgid":"4.f","version":"55'115","reported_seq":150,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.020003+0000","last_change":"2024-04-16T07:49:22.892700+0000","last_active":"2024-04-16T07:49:36.020003+0000","last_peered":"2024-04-16T07:49:36.020003+0000","last_clean":"2024-04-16T07:49:36.020003+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:36.020003+0000","last_undegraded":"2024-04-16T07:49:36.020003+0000","last_fullsized":"2024-04-16T07:49:36.020003+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":193,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.540096+0000","last_change":"2024-04-16T07:49:36.541161+0000","last_active":"2024-04-16T07:49:36.540096+0000","last_peered":"2024-04-16T07:49:36.540096+0000","last_clean":"2024-04-16T07:49:02.265533+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:36.540096+0000","last_undegraded":"2024-04-16T07:49:36.540096+0000","last_fullsized":"2024-04-16T07:49:36.540096+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'101","reported_seq":109,"reported_epoch":55,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:36.137708+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:36.137708+0000","last_peered":"2024-04-16T07:49:36.137708+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:36.137708+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:36.137708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:49:21.822703+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:49:22.788948+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":152,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.495621+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:36.495621+0000","last_peered":"2024-04-16T07:49:36.495621+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:36.495621+0000","last_undegraded":"2024-04-16T07:49:36.495621+0000","last_fullsized":"2024-04-16T07:49:36.495621+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'126","reported_seq":126,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.170181+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:36.170181+0000","last_peered":"2024-04-16T07:49:36.170181+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:36.170181+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:36.170181+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"53'110","reported_seq":107,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:35.979748+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:35.979748+0000","last_peered":"2024-04-16T07:49:35.979748+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:35.979748+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:35.979748+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":124,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:35.984828+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:35.984828+0000","last_peered":"2024-04-16T07:49:35.984828+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:35.984828+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:35.984828+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":231,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.852687+0000","last_change":"2024-04-16T07:49:22.853950+0000","last_active":"2024-04-16T07:49:22.852687+0000","last_peered":"2024-04-16T07:49:22.852687+0000","last_clean":"2024-04-16T07:48:22.913279+0000","last_became_active":"2024-04-16T07:49:22.847179+0000","last_became_peered":"2024-04-16T07:49:22.847179+0000","last_unstale":"2024-04-16T07:49:22.852687+0000","last_undegraded":"2024-04-16T07:49:22.852687+0000","last_fullsized":"2024-04-16T07:49:22.852687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.971860+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:35.971860+0000","last_peered":"2024-04-16T07:49:35.971860+0000","last_clean":"2024-04-16T07:48:22.915206+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:35.971860+0000","last_undegraded":"2024-04-16T07:49:35.971860+0000","last_fullsized":"2024-04-16T07:49:35.971860+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"48'105","reported_seq":127,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.847019+0000","last_change":"2024-04-16T07:49:22.848219+0000","last_active":"2024-04-16T07:49:22.847019+0000","last_peered":"2024-04-16T07:49:22.847019+0000","last_clean":"2024-04-16T07:49:02.322013+0000","last_became_active":"2024-04-16T07:49:22.843784+0000","last_became_peered":"2024-04-16T07:49:22.843784+0000","last_unstale":"2024-04-16T07:49:22.847019+0000","last_undegraded":"2024-04-16T07:49:22.847019+0000","last_fullsized":"2024-04-16T07:49:22.847019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:49:22.790566+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":154,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.772504+0000","last_change":"2024-04-16T07:49:22.862015+0000","last_active":"2024-04-16T07:49:35.772504+0000","last_peered":"2024-04-16T07:49:35.772504+0000","last_clean":"2024-04-16T07:49:35.772504+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:35.772504+0000","last_undegraded":"2024-04-16T07:49:35.772504+0000","last_fullsized":"2024-04-16T07:49:35.772504+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.897013+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:35.897013+0000","last_peered":"2024-04-16T07:49:35.897013+0000","last_clean":"2024-04-16T07:47:55.842517+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:35.897013+0000","last_undegraded":"2024-04-16T07:49:35.897013+0000","last_fullsized":"2024-04-16T07:49:35.897013+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'119","reported_seq":156,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.078781+0000","last_change":"2024-04-16T07:49:36.046350+0000","last_active":"2024-04-16T07:49:36.078781+0000","last_peered":"2024-04-16T07:49:36.078781+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:36.078781+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:36.078781+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":107},{"shards":"2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.801564+0000","last_change":"2024-04-16T07:48:47.921810+0000","last_active":"2024-04-16T07:49:22.801564+0000","last_peered":"2024-04-16T07:49:22.801564+0000","last_clean":"2024-04-16T07:49:22.801564+0000","last_became_active":"2024-04-16T07:48:47.913008+0000","last_became_peered":"2024-04-16T07:48:47.913008+0000","last_unstale":"2024-04-16T07:49:22.801564+0000","last_undegraded":"2024-04-16T07:49:22.801564+0000","last_fullsized":"2024-04-16T07:49:22.801564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":134,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.151363+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:36.151363+0000","last_peered":"2024-04-16T07:49:36.151363+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:36.151363+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:36.151363+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.770984+0000","last_change":"2024-04-16T07:49:22.857045+0000","last_active":"2024-04-16T07:49:35.770984+0000","last_peered":"2024-04-16T07:49:35.770984+0000","last_clean":"2024-04-16T07:49:35.770984+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:35.770984+0000","last_undegraded":"2024-04-16T07:49:35.770984+0000","last_fullsized":"2024-04-16T07:49:35.770984+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:49:22.787540+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.864951+0000","last_change":"2024-04-16T07:49:22.866332+0000","last_active":"2024-04-16T07:49:22.864951+0000","last_peered":"2024-04-16T07:49:22.864951+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:49:22.858691+0000","last_became_peered":"2024-04-16T07:49:22.858691+0000","last_unstale":"2024-04-16T07:49:22.864951+0000","last_undegraded":"2024-04-16T07:49:22.864951+0000","last_fullsized":"2024-04-16T07:49:22.864951+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:49:21.774567+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.794837+0000","last_change":"2024-04-16T07:49:22.896203+0000","last_active":"2024-04-16T07:49:35.794837+0000","last_peered":"2024-04-16T07:49:35.794837+0000","last_clean":"2024-04-16T07:49:35.794837+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:35.794837+0000","last_undegraded":"2024-04-16T07:49:35.794837+0000","last_fullsized":"2024-04-16T07:49:35.794837+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":122,"reported_epoch":55,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:36.034266+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:36.034266+0000","last_peered":"2024-04-16T07:49:36.034266+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:36.034266+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:36.034266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.960858+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:35.960858+0000","last_peered":"2024-04-16T07:49:35.960858+0000","last_clean":"2024-04-16T07:48:22.871937+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:35.960858+0000","last_undegraded":"2024-04-16T07:49:35.960858+0000","last_fullsized":"2024-04-16T07:49:35.960858+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":71303168,"num_objects":1088,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1088,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1809,"ondisk_log_size":1809,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182583,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32624,"kb_used_data":5288,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":5414912,"data_stored":79083184,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928233998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38240,"kb_used_data":5784,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333600,"statfs":{"total":96636764160,"available":96597606400,"internally_reserved":0,"allocated":5922816,"data_stored":87602320,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:53.505 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:53.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:53.613+0000 7f2cfe26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:53.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:53.621+0000 7f2cfe26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:53.808 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:49:53.809 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:49:53.821 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-04-16T07:49:53.173824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71893536,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1225,"num_write_kb":71526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1906,"ondisk_log_size":1906,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145608,"kb_used_data":21864,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341752,"statfs":{"total":386547056640,"available":386397954048,"internally_reserved":0,"allocated":22388736,"data_stored":329411380,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.078782"},"pg_stats":[{"pgid":"4.f","version":"55'115","reported_seq":150,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.020003+0000","last_change":"2024-04-16T07:49:22.892700+0000","last_active":"2024-04-16T07:49:36.020003+0000","last_peered":"2024-04-16T07:49:36.020003+0000","last_clean":"2024-04-16T07:49:36.020003+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:36.020003+0000","last_undegraded":"2024-04-16T07:49:36.020003+0000","last_fullsized":"2024-04-16T07:49:36.020003+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":193,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.540096+0000","last_change":"2024-04-16T07:49:36.541161+0000","last_active":"2024-04-16T07:49:36.540096+0000","last_peered":"2024-04-16T07:49:36.540096+0000","last_clean":"2024-04-16T07:49:02.265533+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:36.540096+0000","last_undegraded":"2024-04-16T07:49:36.540096+0000","last_fullsized":"2024-04-16T07:49:36.540096+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'101","reported_seq":109,"reported_epoch":55,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:36.137708+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:36.137708+0000","last_peered":"2024-04-16T07:49:36.137708+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:36.137708+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:36.137708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:49:21.822703+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:49:22.788948+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":152,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.495621+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:36.495621+0000","last_peered":"2024-04-16T07:49:36.495621+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:36.495621+0000","last_undegraded":"2024-04-16T07:49:36.495621+0000","last_fullsized":"2024-04-16T07:49:36.495621+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'126","reported_seq":126,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.170181+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:36.170181+0000","last_peered":"2024-04-16T07:49:36.170181+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:36.170181+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:36.170181+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"53'110","reported_seq":107,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:35.979748+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:35.979748+0000","last_peered":"2024-04-16T07:49:35.979748+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:35.979748+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:35.979748+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":124,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:35.984828+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:35.984828+0000","last_peered":"2024-04-16T07:49:35.984828+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:35.984828+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:35.984828+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":231,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.852687+0000","last_change":"2024-04-16T07:49:22.853950+0000","last_active":"2024-04-16T07:49:22.852687+0000","last_peered":"2024-04-16T07:49:22.852687+0000","last_clean":"2024-04-16T07:48:22.913279+0000","last_became_active":"2024-04-16T07:49:22.847179+0000","last_became_peered":"2024-04-16T07:49:22.847179+0000","last_unstale":"2024-04-16T07:49:22.852687+0000","last_undegraded":"2024-04-16T07:49:22.852687+0000","last_fullsized":"2024-04-16T07:49:22.852687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.971860+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:35.971860+0000","last_peered":"2024-04-16T07:49:35.971860+0000","last_clean":"2024-04-16T07:48:22.915206+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:35.971860+0000","last_undegraded":"2024-04-16T07:49:35.971860+0000","last_fullsized":"2024-04-16T07:49:35.971860+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"48'105","reported_seq":127,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.847019+0000","last_change":"2024-04-16T07:49:22.848219+0000","last_active":"2024-04-16T07:49:22.847019+0000","last_peered":"2024-04-16T07:49:22.847019+0000","last_clean":"2024-04-16T07:49:02.322013+0000","last_became_active":"2024-04-16T07:49:22.843784+0000","last_became_peered":"2024-04-16T07:49:22.843784+0000","last_unstale":"2024-04-16T07:49:22.847019+0000","last_undegraded":"2024-04-16T07:49:22.847019+0000","last_fullsized":"2024-04-16T07:49:22.847019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:49:22.790566+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":154,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.772504+0000","last_change":"2024-04-16T07:49:22.862015+0000","last_active":"2024-04-16T07:49:35.772504+0000","last_peered":"2024-04-16T07:49:35.772504+0000","last_clean":"2024-04-16T07:49:35.772504+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:35.772504+0000","last_undegraded":"2024-04-16T07:49:35.772504+0000","last_fullsized":"2024-04-16T07:49:35.772504+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.897013+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:35.897013+0000","last_peered":"2024-04-16T07:49:35.897013+0000","last_clean":"2024-04-16T07:47:55.842517+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:35.897013+0000","last_undegraded":"2024-04-16T07:49:35.897013+0000","last_fullsized":"2024-04-16T07:49:35.897013+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'119","reported_seq":156,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.078781+0000","last_change":"2024-04-16T07:49:36.046350+0000","last_active":"2024-04-16T07:49:36.078781+0000","last_peered":"2024-04-16T07:49:36.078781+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:36.078781+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:36.078781+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":107},{"shards":"2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.801564+0000","last_change":"2024-04-16T07:48:47.921810+0000","last_active":"2024-04-16T07:49:22.801564+0000","last_peered":"2024-04-16T07:49:22.801564+0000","last_clean":"2024-04-16T07:49:22.801564+0000","last_became_active":"2024-04-16T07:48:47.913008+0000","last_became_peered":"2024-04-16T07:48:47.913008+0000","last_unstale":"2024-04-16T07:49:22.801564+0000","last_undegraded":"2024-04-16T07:49:22.801564+0000","last_fullsized":"2024-04-16T07:49:22.801564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":134,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.151363+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:36.151363+0000","last_peered":"2024-04-16T07:49:36.151363+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:36.151363+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:36.151363+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.770984+0000","last_change":"2024-04-16T07:49:22.857045+0000","last_active":"2024-04-16T07:49:35.770984+0000","last_peered":"2024-04-16T07:49:35.770984+0000","last_clean":"2024-04-16T07:49:35.770984+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:35.770984+0000","last_undegraded":"2024-04-16T07:49:35.770984+0000","last_fullsized":"2024-04-16T07:49:35.770984+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:49:22.787540+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.864951+0000","last_change":"2024-04-16T07:49:22.866332+0000","last_active":"2024-04-16T07:49:22.864951+0000","last_peered":"2024-04-16T07:49:22.864951+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:49:22.858691+0000","last_became_peered":"2024-04-16T07:49:22.858691+0000","last_unstale":"2024-04-16T07:49:22.864951+0000","last_undegraded":"2024-04-16T07:49:22.864951+0000","last_fullsized":"2024-04-16T07:49:22.864951+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:49:21.774567+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.794837+0000","last_change":"2024-04-16T07:49:22.896203+0000","last_active":"2024-04-16T07:49:35.794837+0000","last_peered":"2024-04-16T07:49:35.794837+0000","last_clean":"2024-04-16T07:49:35.794837+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:35.794837+0000","last_undegraded":"2024-04-16T07:49:35.794837+0000","last_fullsized":"2024-04-16T07:49:35.794837+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":122,"reported_epoch":55,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:36.034266+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:36.034266+0000","last_peered":"2024-04-16T07:49:36.034266+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:36.034266+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:36.034266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.960858+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:35.960858+0000","last_peered":"2024-04-16T07:49:35.960858+0000","last_clean":"2024-04-16T07:48:22.871937+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:35.960858+0000","last_undegraded":"2024-04-16T07:49:35.960858+0000","last_fullsized":"2024-04-16T07:49:35.960858+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":71303168,"num_objects":1088,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1088,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1809,"ondisk_log_size":1809,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182583,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32624,"kb_used_data":5288,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":5414912,"data_stored":79083184,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928233998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38240,"kb_used_data":5784,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333600,"statfs":{"total":96636764160,"available":96597606400,"internally_reserved":0,"allocated":5922816,"data_stored":87602320,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:53.822 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:49:53.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:49:53.878680+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.348198 2024-04-16T07:49:53.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:49:53.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 60 16 1934 1918 1.99767 0 - 0.348198 2024-04-16T07:49:53.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:53.930+0000 7f9a82aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:53.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:53.937+0000 7f9a82aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:54.191 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:49:54.191 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 146/3270 objects degraded (4.465%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":271,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1713253774,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+degraded","count":5},{"state_name":"peering","count":2},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":25,"num_pools":3,"num_objects":1090,"data_bytes":71893536,"bytes_used":149102592,"bytes_avail":386397954048,"bytes_total":386547056640,"inactive_pgs_ratio":0.079999998211860657,"degraded_objects":146,"degraded_total":3270,"degraded_ratio":0.04464831804281346},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:49:54.204 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:49:54.204 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:54.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:54.313+0000 7f705ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:54.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:54.321+0000 7f705ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:54.504 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:49:54.504 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:49:54.517 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-04-16T07:49:53.173824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71893536,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1225,"num_write_kb":71526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1906,"ondisk_log_size":1906,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145608,"kb_used_data":21864,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341752,"statfs":{"total":386547056640,"available":386397954048,"internally_reserved":0,"allocated":22388736,"data_stored":329411380,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.078782"},"pg_stats":[{"pgid":"4.f","version":"55'115","reported_seq":150,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.020003+0000","last_change":"2024-04-16T07:49:22.892700+0000","last_active":"2024-04-16T07:49:36.020003+0000","last_peered":"2024-04-16T07:49:36.020003+0000","last_clean":"2024-04-16T07:49:36.020003+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:36.020003+0000","last_undegraded":"2024-04-16T07:49:36.020003+0000","last_fullsized":"2024-04-16T07:49:36.020003+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":193,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.540096+0000","last_change":"2024-04-16T07:49:36.541161+0000","last_active":"2024-04-16T07:49:36.540096+0000","last_peered":"2024-04-16T07:49:36.540096+0000","last_clean":"2024-04-16T07:49:02.265533+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:36.540096+0000","last_undegraded":"2024-04-16T07:49:36.540096+0000","last_fullsized":"2024-04-16T07:49:36.540096+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'101","reported_seq":109,"reported_epoch":55,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:36.137708+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:36.137708+0000","last_peered":"2024-04-16T07:49:36.137708+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:36.137708+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:36.137708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:49:21.822703+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:49:22.788948+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":152,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:36.495621+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:36.495621+0000","last_peered":"2024-04-16T07:49:36.495621+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:36.495621+0000","last_undegraded":"2024-04-16T07:49:36.495621+0000","last_fullsized":"2024-04-16T07:49:36.495621+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'126","reported_seq":126,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.170181+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:36.170181+0000","last_peered":"2024-04-16T07:49:36.170181+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:36.170181+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:36.170181+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"53'110","reported_seq":107,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:35.979748+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:35.979748+0000","last_peered":"2024-04-16T07:49:35.979748+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:35.979748+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:35.979748+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":124,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:35.984828+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:35.984828+0000","last_peered":"2024-04-16T07:49:35.984828+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:35.984828+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:35.984828+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":231,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.852687+0000","last_change":"2024-04-16T07:49:22.853950+0000","last_active":"2024-04-16T07:49:22.852687+0000","last_peered":"2024-04-16T07:49:22.852687+0000","last_clean":"2024-04-16T07:48:22.913279+0000","last_became_active":"2024-04-16T07:49:22.847179+0000","last_became_peered":"2024-04-16T07:49:22.847179+0000","last_unstale":"2024-04-16T07:49:22.852687+0000","last_undegraded":"2024-04-16T07:49:22.852687+0000","last_fullsized":"2024-04-16T07:49:22.852687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.971860+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:35.971860+0000","last_peered":"2024-04-16T07:49:35.971860+0000","last_clean":"2024-04-16T07:48:22.915206+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:35.971860+0000","last_undegraded":"2024-04-16T07:49:35.971860+0000","last_fullsized":"2024-04-16T07:49:35.971860+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"48'105","reported_seq":127,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.847019+0000","last_change":"2024-04-16T07:49:22.848219+0000","last_active":"2024-04-16T07:49:22.847019+0000","last_peered":"2024-04-16T07:49:22.847019+0000","last_clean":"2024-04-16T07:49:02.322013+0000","last_became_active":"2024-04-16T07:49:22.843784+0000","last_became_peered":"2024-04-16T07:49:22.843784+0000","last_unstale":"2024-04-16T07:49:22.847019+0000","last_undegraded":"2024-04-16T07:49:22.847019+0000","last_fullsized":"2024-04-16T07:49:22.847019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:49:22.790566+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":154,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.772504+0000","last_change":"2024-04-16T07:49:22.862015+0000","last_active":"2024-04-16T07:49:35.772504+0000","last_peered":"2024-04-16T07:49:35.772504+0000","last_clean":"2024-04-16T07:49:35.772504+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:35.772504+0000","last_undegraded":"2024-04-16T07:49:35.772504+0000","last_fullsized":"2024-04-16T07:49:35.772504+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.897013+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:35.897013+0000","last_peered":"2024-04-16T07:49:35.897013+0000","last_clean":"2024-04-16T07:47:55.842517+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:35.897013+0000","last_undegraded":"2024-04-16T07:49:35.897013+0000","last_fullsized":"2024-04-16T07:49:35.897013+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'119","reported_seq":156,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.078781+0000","last_change":"2024-04-16T07:49:36.046350+0000","last_active":"2024-04-16T07:49:36.078781+0000","last_peered":"2024-04-16T07:49:36.078781+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:36.078781+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:36.078781+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":107},{"shards":"2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.801564+0000","last_change":"2024-04-16T07:48:47.921810+0000","last_active":"2024-04-16T07:49:22.801564+0000","last_peered":"2024-04-16T07:49:22.801564+0000","last_clean":"2024-04-16T07:49:22.801564+0000","last_became_active":"2024-04-16T07:48:47.913008+0000","last_became_peered":"2024-04-16T07:48:47.913008+0000","last_unstale":"2024-04-16T07:49:22.801564+0000","last_undegraded":"2024-04-16T07:49:22.801564+0000","last_fullsized":"2024-04-16T07:49:22.801564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":134,"reported_epoch":55,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:36.151363+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:36.151363+0000","last_peered":"2024-04-16T07:49:36.151363+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:36.151363+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:36.151363+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.770984+0000","last_change":"2024-04-16T07:49:22.857045+0000","last_active":"2024-04-16T07:49:35.770984+0000","last_peered":"2024-04-16T07:49:35.770984+0000","last_clean":"2024-04-16T07:49:35.770984+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:35.770984+0000","last_undegraded":"2024-04-16T07:49:35.770984+0000","last_fullsized":"2024-04-16T07:49:35.770984+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:49:22.787540+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.864951+0000","last_change":"2024-04-16T07:49:22.866332+0000","last_active":"2024-04-16T07:49:22.864951+0000","last_peered":"2024-04-16T07:49:22.864951+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:49:22.858691+0000","last_became_peered":"2024-04-16T07:49:22.858691+0000","last_unstale":"2024-04-16T07:49:22.864951+0000","last_undegraded":"2024-04-16T07:49:22.864951+0000","last_fullsized":"2024-04-16T07:49:22.864951+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:49:21.774567+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.794837+0000","last_change":"2024-04-16T07:49:22.896203+0000","last_active":"2024-04-16T07:49:35.794837+0000","last_peered":"2024-04-16T07:49:35.794837+0000","last_clean":"2024-04-16T07:49:35.794837+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:35.794837+0000","last_undegraded":"2024-04-16T07:49:35.794837+0000","last_fullsized":"2024-04-16T07:49:35.794837+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":122,"reported_epoch":55,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:36.034266+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:36.034266+0000","last_peered":"2024-04-16T07:49:36.034266+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:36.034266+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:36.034266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-16T07:49:35.960858+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:35.960858+0000","last_peered":"2024-04-16T07:49:35.960858+0000","last_clean":"2024-04-16T07:48:22.871937+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:35.960858+0000","last_undegraded":"2024-04-16T07:49:35.960858+0000","last_fullsized":"2024-04-16T07:49:35.960858+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":71303168,"num_objects":1088,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1088,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1809,"ondisk_log_size":1809,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182583,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32624,"kb_used_data":5288,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":5414912,"data_stored":79083184,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928233998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38240,"kb_used_data":5784,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333600,"statfs":{"total":96636764160,"available":96597606400,"internally_reserved":0,"allocated":5922816,"data_stored":87602320,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:54.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 61 16 1934 1918 1.96492 0 - 0.348198 2024-04-16T07:49:55.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2024-04-16T07:49:55.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~16s 2024-04-16T07:49:55.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 62 16 1934 1918 1.93322 0 - 0.348198 2024-04-16T07:49:56.878 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 63 16 1934 1918 1.90254 0 - 0.348198 2024-04-16T07:49:57.518 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:57.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:57.640+0000 7ff18ae0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:57.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:57.648+0000 7ff18ae0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:49:57.835 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:49:57.848 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-04-16T07:49:57.097253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71893536,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1225,"num_write_kb":71526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1911,"ondisk_log_size":1911,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145664,"kb_used_data":21920,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341696,"statfs":{"total":386547056640,"available":386397896704,"internally_reserved":0,"allocated":22446080,"data_stored":329434095,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001671"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.201497+0000","last_change":"2024-04-16T07:49:22.892700+0000","last_active":"2024-04-16T07:49:53.201497+0000","last_peered":"2024-04-16T07:49:53.201497+0000","last_clean":"2024-04-16T07:49:53.201497+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:53.201497+0000","last_undegraded":"2024-04-16T07:49:53.201497+0000","last_fullsized":"2024-04-16T07:49:53.201497+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":205,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.198548+0000","last_change":"2024-04-16T07:49:36.541161+0000","last_active":"2024-04-16T07:49:53.198548+0000","last_peered":"2024-04-16T07:49:53.198548+0000","last_clean":"2024-04-16T07:49:53.198548+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:53.198548+0000","last_undegraded":"2024-04-16T07:49:53.198548+0000","last_fullsized":"2024-04-16T07:49:53.198548+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":116,"reported_epoch":58,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:52.183587+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:52.183587+0000","last_peered":"2024-04-16T07:49:52.183587+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:52.183587+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:52.183587+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:49:21.822703+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:49:22.788948+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":159,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-16T07:49:52.196133+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:52.196133+0000","last_peered":"2024-04-16T07:49:52.196133+0000","last_clean":"2024-04-16T07:49:52.196133+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:52.196133+0000","last_undegraded":"2024-04-16T07:49:52.196133+0000","last_fullsized":"2024-04-16T07:49:52.196133+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":133,"reported_epoch":58,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:52.191930+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:52.191930+0000","last_peered":"2024-04-16T07:49:52.191930+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:52.191930+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:52.191930+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":114,"reported_epoch":58,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:52.206243+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:52.206243+0000","last_peered":"2024-04-16T07:49:52.206243+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:52.206243+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:52.206243+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":131,"reported_epoch":58,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:52.194509+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:52.194509+0000","last_peered":"2024-04-16T07:49:52.194509+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:52.194509+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:52.194509+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":231,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.852687+0000","last_change":"2024-04-16T07:49:22.853950+0000","last_active":"2024-04-16T07:49:22.852687+0000","last_peered":"2024-04-16T07:49:22.852687+0000","last_clean":"2024-04-16T07:48:22.913279+0000","last_became_active":"2024-04-16T07:49:22.847179+0000","last_became_peered":"2024-04-16T07:49:22.847179+0000","last_unstale":"2024-04-16T07:49:22.852687+0000","last_undegraded":"2024-04-16T07:49:22.852687+0000","last_fullsized":"2024-04-16T07:49:22.852687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":125,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-16T07:49:52.181861+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:52.181861+0000","last_peered":"2024-04-16T07:49:52.181861+0000","last_clean":"2024-04-16T07:49:52.181861+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:52.181861+0000","last_undegraded":"2024-04-16T07:49:52.181861+0000","last_fullsized":"2024-04-16T07:49:52.181861+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"48'105","reported_seq":127,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.847019+0000","last_change":"2024-04-16T07:49:22.848219+0000","last_active":"2024-04-16T07:49:22.847019+0000","last_peered":"2024-04-16T07:49:22.847019+0000","last_clean":"2024-04-16T07:49:02.322013+0000","last_became_active":"2024-04-16T07:49:22.843784+0000","last_became_peered":"2024-04-16T07:49:22.843784+0000","last_unstale":"2024-04-16T07:49:22.847019+0000","last_undegraded":"2024-04-16T07:49:22.847019+0000","last_fullsized":"2024-04-16T07:49:22.847019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:49:22.790566+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":166,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.189582+0000","last_change":"2024-04-16T07:49:22.862015+0000","last_active":"2024-04-16T07:49:53.189582+0000","last_peered":"2024-04-16T07:49:53.189582+0000","last_clean":"2024-04-16T07:49:53.189582+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:53.189582+0000","last_undegraded":"2024-04-16T07:49:53.189582+0000","last_fullsized":"2024-04-16T07:49:53.189582+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":83,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-16T07:49:52.185827+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:52.185827+0000","last_peered":"2024-04-16T07:49:52.185827+0000","last_clean":"2024-04-16T07:49:52.185827+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:52.185827+0000","last_undegraded":"2024-04-16T07:49:52.185827+0000","last_fullsized":"2024-04-16T07:49:52.185827+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":168,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:53.195326+0000","last_change":"2024-04-16T07:49:36.046350+0000","last_active":"2024-04-16T07:49:53.195326+0000","last_peered":"2024-04-16T07:49:53.195326+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:53.195326+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:53.195326+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":107},{"shards":"2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.801564+0000","last_change":"2024-04-16T07:48:47.921810+0000","last_active":"2024-04-16T07:49:22.801564+0000","last_peered":"2024-04-16T07:49:22.801564+0000","last_clean":"2024-04-16T07:49:22.801564+0000","last_became_active":"2024-04-16T07:48:47.913008+0000","last_became_peered":"2024-04-16T07:48:47.913008+0000","last_unstale":"2024-04-16T07:49:22.801564+0000","last_undegraded":"2024-04-16T07:49:22.801564+0000","last_fullsized":"2024-04-16T07:49:22.801564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":141,"reported_epoch":58,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:52.203616+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:52.203616+0000","last_peered":"2024-04-16T07:49:52.203616+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:52.203616+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:52.203616+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.192178+0000","last_change":"2024-04-16T07:49:22.857045+0000","last_active":"2024-04-16T07:49:53.192178+0000","last_peered":"2024-04-16T07:49:53.192178+0000","last_clean":"2024-04-16T07:49:53.192178+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:53.192178+0000","last_undegraded":"2024-04-16T07:49:53.192178+0000","last_fullsized":"2024-04-16T07:49:53.192178+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:49:22.787540+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.864951+0000","last_change":"2024-04-16T07:49:22.866332+0000","last_active":"2024-04-16T07:49:22.864951+0000","last_peered":"2024-04-16T07:49:22.864951+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:49:22.858691+0000","last_became_peered":"2024-04-16T07:49:22.858691+0000","last_unstale":"2024-04-16T07:49:22.864951+0000","last_undegraded":"2024-04-16T07:49:22.864951+0000","last_fullsized":"2024-04-16T07:49:22.864951+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:49:21.774567+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.210443+0000","last_change":"2024-04-16T07:49:22.896203+0000","last_active":"2024-04-16T07:49:53.210443+0000","last_peered":"2024-04-16T07:49:53.210443+0000","last_clean":"2024-04-16T07:49:53.210443+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:53.210443+0000","last_undegraded":"2024-04-16T07:49:53.210443+0000","last_fullsized":"2024-04-16T07:49:53.210443+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":129,"reported_epoch":58,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:52.199416+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:52.199416+0000","last_peered":"2024-04-16T07:49:52.199416+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:52.199416+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:52.199416+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-16T07:49:52.200801+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:52.200801+0000","last_peered":"2024-04-16T07:49:52.200801+0000","last_clean":"2024-04-16T07:49:52.200801+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:52.200801+0000","last_undegraded":"2024-04-16T07:49:52.200801+0000","last_fullsized":"2024-04-16T07:49:52.200801+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":71303168,"num_objects":1088,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1088,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1814,"ondisk_log_size":1814,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182588,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32656,"kb_used_data":5320,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":5447680,"data_stored":79096164,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38264,"kb_used_data":5808,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333576,"statfs":{"total":96636764160,"available":96597581824,"internally_reserved":0,"allocated":5947392,"data_stored":87612055,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:57.849 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:49:57.877 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 64 16 1934 1918 1.87281 0 - 0.348198 2024-04-16T07:49:57.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:57.957+0000 7f32bf46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:57.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:57.964+0000 7f32bf46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:58.217 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:49:58.218 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 146/3270 objects degraded (4.465%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":275,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1713253774,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+degraded","count":5},{"state_name":"peering","count":2},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":25,"num_pools":3,"num_objects":1090,"data_bytes":71893536,"bytes_used":149159936,"bytes_avail":386397896704,"bytes_total":386547056640,"inactive_pgs_ratio":0.079999998211860657,"degraded_objects":146,"degraded_total":3270,"degraded_ratio":0.04464831804281346},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:49:58.231 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:49:58.231 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:58.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:58.380+0000 7f5c60a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:58.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:49:58.387+0000 7f5c60a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:58.572 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:49:58.573 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:49:58.585 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-04-16T07:49:57.097253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71893536,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1225,"num_write_kb":71526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1911,"ondisk_log_size":1911,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145664,"kb_used_data":21920,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341696,"statfs":{"total":386547056640,"available":386397896704,"internally_reserved":0,"allocated":22446080,"data_stored":329434095,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001671"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.201497+0000","last_change":"2024-04-16T07:49:22.892700+0000","last_active":"2024-04-16T07:49:53.201497+0000","last_peered":"2024-04-16T07:49:53.201497+0000","last_clean":"2024-04-16T07:49:53.201497+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:53.201497+0000","last_undegraded":"2024-04-16T07:49:53.201497+0000","last_fullsized":"2024-04-16T07:49:53.201497+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":205,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.198548+0000","last_change":"2024-04-16T07:49:36.541161+0000","last_active":"2024-04-16T07:49:53.198548+0000","last_peered":"2024-04-16T07:49:53.198548+0000","last_clean":"2024-04-16T07:49:53.198548+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:53.198548+0000","last_undegraded":"2024-04-16T07:49:53.198548+0000","last_fullsized":"2024-04-16T07:49:53.198548+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":116,"reported_epoch":58,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:52.183587+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:52.183587+0000","last_peered":"2024-04-16T07:49:52.183587+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:52.183587+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:52.183587+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:49:21.822703+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:49:22.788948+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":159,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-16T07:49:52.196133+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:52.196133+0000","last_peered":"2024-04-16T07:49:52.196133+0000","last_clean":"2024-04-16T07:49:52.196133+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:52.196133+0000","last_undegraded":"2024-04-16T07:49:52.196133+0000","last_fullsized":"2024-04-16T07:49:52.196133+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":133,"reported_epoch":58,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:52.191930+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:52.191930+0000","last_peered":"2024-04-16T07:49:52.191930+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:52.191930+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:52.191930+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":114,"reported_epoch":58,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:52.206243+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:52.206243+0000","last_peered":"2024-04-16T07:49:52.206243+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:52.206243+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:52.206243+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":131,"reported_epoch":58,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:52.194509+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:52.194509+0000","last_peered":"2024-04-16T07:49:52.194509+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:52.194509+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:52.194509+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":231,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.852687+0000","last_change":"2024-04-16T07:49:22.853950+0000","last_active":"2024-04-16T07:49:22.852687+0000","last_peered":"2024-04-16T07:49:22.852687+0000","last_clean":"2024-04-16T07:48:22.913279+0000","last_became_active":"2024-04-16T07:49:22.847179+0000","last_became_peered":"2024-04-16T07:49:22.847179+0000","last_unstale":"2024-04-16T07:49:22.852687+0000","last_undegraded":"2024-04-16T07:49:22.852687+0000","last_fullsized":"2024-04-16T07:49:22.852687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":125,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-16T07:49:52.181861+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:52.181861+0000","last_peered":"2024-04-16T07:49:52.181861+0000","last_clean":"2024-04-16T07:49:52.181861+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:52.181861+0000","last_undegraded":"2024-04-16T07:49:52.181861+0000","last_fullsized":"2024-04-16T07:49:52.181861+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"48'105","reported_seq":127,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.847019+0000","last_change":"2024-04-16T07:49:22.848219+0000","last_active":"2024-04-16T07:49:22.847019+0000","last_peered":"2024-04-16T07:49:22.847019+0000","last_clean":"2024-04-16T07:49:02.322013+0000","last_became_active":"2024-04-16T07:49:22.843784+0000","last_became_peered":"2024-04-16T07:49:22.843784+0000","last_unstale":"2024-04-16T07:49:22.847019+0000","last_undegraded":"2024-04-16T07:49:22.847019+0000","last_fullsized":"2024-04-16T07:49:22.847019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:49:22.790566+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":166,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.189582+0000","last_change":"2024-04-16T07:49:22.862015+0000","last_active":"2024-04-16T07:49:53.189582+0000","last_peered":"2024-04-16T07:49:53.189582+0000","last_clean":"2024-04-16T07:49:53.189582+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:53.189582+0000","last_undegraded":"2024-04-16T07:49:53.189582+0000","last_fullsized":"2024-04-16T07:49:53.189582+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":83,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-16T07:49:52.185827+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:52.185827+0000","last_peered":"2024-04-16T07:49:52.185827+0000","last_clean":"2024-04-16T07:49:52.185827+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:52.185827+0000","last_undegraded":"2024-04-16T07:49:52.185827+0000","last_fullsized":"2024-04-16T07:49:52.185827+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":168,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:53.195326+0000","last_change":"2024-04-16T07:49:36.046350+0000","last_active":"2024-04-16T07:49:53.195326+0000","last_peered":"2024-04-16T07:49:53.195326+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:53.195326+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:53.195326+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":107},{"shards":"2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.801564+0000","last_change":"2024-04-16T07:48:47.921810+0000","last_active":"2024-04-16T07:49:22.801564+0000","last_peered":"2024-04-16T07:49:22.801564+0000","last_clean":"2024-04-16T07:49:22.801564+0000","last_became_active":"2024-04-16T07:48:47.913008+0000","last_became_peered":"2024-04-16T07:48:47.913008+0000","last_unstale":"2024-04-16T07:49:22.801564+0000","last_undegraded":"2024-04-16T07:49:22.801564+0000","last_fullsized":"2024-04-16T07:49:22.801564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":141,"reported_epoch":58,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:52.203616+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:52.203616+0000","last_peered":"2024-04-16T07:49:52.203616+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:52.203616+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:52.203616+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.192178+0000","last_change":"2024-04-16T07:49:22.857045+0000","last_active":"2024-04-16T07:49:53.192178+0000","last_peered":"2024-04-16T07:49:53.192178+0000","last_clean":"2024-04-16T07:49:53.192178+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:53.192178+0000","last_undegraded":"2024-04-16T07:49:53.192178+0000","last_fullsized":"2024-04-16T07:49:53.192178+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:49:22.787540+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.864951+0000","last_change":"2024-04-16T07:49:22.866332+0000","last_active":"2024-04-16T07:49:22.864951+0000","last_peered":"2024-04-16T07:49:22.864951+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:49:22.858691+0000","last_became_peered":"2024-04-16T07:49:22.858691+0000","last_unstale":"2024-04-16T07:49:22.864951+0000","last_undegraded":"2024-04-16T07:49:22.864951+0000","last_fullsized":"2024-04-16T07:49:22.864951+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:49:21.774567+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.210443+0000","last_change":"2024-04-16T07:49:22.896203+0000","last_active":"2024-04-16T07:49:53.210443+0000","last_peered":"2024-04-16T07:49:53.210443+0000","last_clean":"2024-04-16T07:49:53.210443+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:53.210443+0000","last_undegraded":"2024-04-16T07:49:53.210443+0000","last_fullsized":"2024-04-16T07:49:53.210443+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":129,"reported_epoch":58,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:52.199416+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:52.199416+0000","last_peered":"2024-04-16T07:49:52.199416+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:52.199416+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:52.199416+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-16T07:49:52.200801+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:52.200801+0000","last_peered":"2024-04-16T07:49:52.200801+0000","last_clean":"2024-04-16T07:49:52.200801+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:52.200801+0000","last_undegraded":"2024-04-16T07:49:52.200801+0000","last_fullsized":"2024-04-16T07:49:52.200801+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":71303168,"num_objects":1088,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1088,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1814,"ondisk_log_size":1814,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182588,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32656,"kb_used_data":5320,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":5447680,"data_stored":79096164,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38264,"kb_used_data":5808,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333576,"statfs":{"total":96636764160,"available":96597581824,"internally_reserved":0,"allocated":5947392,"data_stored":87612055,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:58.878 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 65 16 1934 1918 1.844 0 - 0.348198 2024-04-16T07:49:59.878 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 66 16 1934 1918 1.81606 0 - 0.348198 2024-04-16T07:50:00.878 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 67 16 1934 1918 1.78895 0 - 0.348198 2024-04-16T07:50:00.996 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~22s 2024-04-16T07:50:00.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~22s 2024-04-16T07:50:01.588 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:01.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:01.711+0000 7f1fc615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:01.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:01.719+0000 7f1fc615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:01.878 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 68 16 1934 1918 1.76265 0 - 0.348198 2024-04-16T07:50:01.903 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:01.904 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:01.916 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-04-16T07:50:01.098311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71893536,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1225,"num_write_kb":71526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1911,"ondisk_log_size":1911,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145672,"kb_used_data":21928,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341688,"statfs":{"total":386547056640,"available":386397888512,"internally_reserved":0,"allocated":22454272,"data_stored":329437340,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.931697"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.201497+0000","last_change":"2024-04-16T07:49:22.892700+0000","last_active":"2024-04-16T07:49:53.201497+0000","last_peered":"2024-04-16T07:49:53.201497+0000","last_clean":"2024-04-16T07:49:53.201497+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:53.201497+0000","last_undegraded":"2024-04-16T07:49:53.201497+0000","last_fullsized":"2024-04-16T07:49:53.201497+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":205,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.198548+0000","last_change":"2024-04-16T07:49:36.541161+0000","last_active":"2024-04-16T07:49:53.198548+0000","last_peered":"2024-04-16T07:49:53.198548+0000","last_clean":"2024-04-16T07:49:53.198548+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:53.198548+0000","last_undegraded":"2024-04-16T07:49:53.198548+0000","last_fullsized":"2024-04-16T07:49:53.198548+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:49:21.822703+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:49:22.788948+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":231,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.852687+0000","last_change":"2024-04-16T07:49:22.853950+0000","last_active":"2024-04-16T07:49:22.852687+0000","last_peered":"2024-04-16T07:49:22.852687+0000","last_clean":"2024-04-16T07:48:22.913279+0000","last_became_active":"2024-04-16T07:49:22.847179+0000","last_became_peered":"2024-04-16T07:49:22.847179+0000","last_unstale":"2024-04-16T07:49:22.852687+0000","last_undegraded":"2024-04-16T07:49:22.852687+0000","last_fullsized":"2024-04-16T07:49:22.852687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"48'105","reported_seq":127,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.847019+0000","last_change":"2024-04-16T07:49:22.848219+0000","last_active":"2024-04-16T07:49:22.847019+0000","last_peered":"2024-04-16T07:49:22.847019+0000","last_clean":"2024-04-16T07:49:02.322013+0000","last_became_active":"2024-04-16T07:49:22.843784+0000","last_became_peered":"2024-04-16T07:49:22.843784+0000","last_unstale":"2024-04-16T07:49:22.847019+0000","last_undegraded":"2024-04-16T07:49:22.847019+0000","last_fullsized":"2024-04-16T07:49:22.847019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:49:22.790566+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":166,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.189582+0000","last_change":"2024-04-16T07:49:22.862015+0000","last_active":"2024-04-16T07:49:53.189582+0000","last_peered":"2024-04-16T07:49:53.189582+0000","last_clean":"2024-04-16T07:49:53.189582+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:53.189582+0000","last_undegraded":"2024-04-16T07:49:53.189582+0000","last_fullsized":"2024-04-16T07:49:53.189582+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":168,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:53.195326+0000","last_change":"2024-04-16T07:49:36.046350+0000","last_active":"2024-04-16T07:49:53.195326+0000","last_peered":"2024-04-16T07:49:53.195326+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:53.195326+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:53.195326+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":107},{"shards":"2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.801564+0000","last_change":"2024-04-16T07:48:47.921810+0000","last_active":"2024-04-16T07:49:22.801564+0000","last_peered":"2024-04-16T07:49:22.801564+0000","last_clean":"2024-04-16T07:49:22.801564+0000","last_became_active":"2024-04-16T07:48:47.913008+0000","last_became_peered":"2024-04-16T07:48:47.913008+0000","last_unstale":"2024-04-16T07:49:22.801564+0000","last_undegraded":"2024-04-16T07:49:22.801564+0000","last_fullsized":"2024-04-16T07:49:22.801564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.192178+0000","last_change":"2024-04-16T07:49:22.857045+0000","last_active":"2024-04-16T07:49:53.192178+0000","last_peered":"2024-04-16T07:49:53.192178+0000","last_clean":"2024-04-16T07:49:53.192178+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:53.192178+0000","last_undegraded":"2024-04-16T07:49:53.192178+0000","last_fullsized":"2024-04-16T07:49:53.192178+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:49:22.787540+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.864951+0000","last_change":"2024-04-16T07:49:22.866332+0000","last_active":"2024-04-16T07:49:22.864951+0000","last_peered":"2024-04-16T07:49:22.864951+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:49:22.858691+0000","last_became_peered":"2024-04-16T07:49:22.858691+0000","last_unstale":"2024-04-16T07:49:22.864951+0000","last_undegraded":"2024-04-16T07:49:22.864951+0000","last_fullsized":"2024-04-16T07:49:22.864951+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:49:21.774567+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.210443+0000","last_change":"2024-04-16T07:49:22.896203+0000","last_active":"2024-04-16T07:49:53.210443+0000","last_peered":"2024-04-16T07:49:53.210443+0000","last_clean":"2024-04-16T07:49:53.210443+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:53.210443+0000","last_undegraded":"2024-04-16T07:49:53.210443+0000","last_fullsized":"2024-04-16T07:49:53.210443+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":71303168,"num_objects":1088,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1088,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1814,"ondisk_log_size":1814,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32656,"kb_used_data":5320,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":5447680,"data_stored":79096164,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:01.917 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:02.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:02.026+0000 7f1083aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:02.033+0000 7f1083aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:02.288 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:02.289 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 146/3270 objects degraded (4.465%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":279,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1713253774,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+degraded","count":5},{"state_name":"peering","count":2},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":25,"num_pools":3,"num_objects":1090,"data_bytes":71893536,"bytes_used":149168128,"bytes_avail":386397888512,"bytes_total":386547056640,"inactive_pgs_ratio":0.079999998211860657,"degraded_objects":146,"degraded_total":3270,"degraded_ratio":0.04464831804281346},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:02.302 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:02.302 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:02.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:02.411+0000 7fac4159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:02.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:02.419+0000 7fac4159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:02.607 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:02.607 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:02.621 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-04-16T07:50:01.098311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71893536,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1225,"num_write_kb":71526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1911,"ondisk_log_size":1911,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145672,"kb_used_data":21928,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341688,"statfs":{"total":386547056640,"available":386397888512,"internally_reserved":0,"allocated":22454272,"data_stored":329437340,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.931697"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.201497+0000","last_change":"2024-04-16T07:49:22.892700+0000","last_active":"2024-04-16T07:49:53.201497+0000","last_peered":"2024-04-16T07:49:53.201497+0000","last_clean":"2024-04-16T07:49:53.201497+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:53.201497+0000","last_undegraded":"2024-04-16T07:49:53.201497+0000","last_fullsized":"2024-04-16T07:49:53.201497+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":205,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.198548+0000","last_change":"2024-04-16T07:49:36.541161+0000","last_active":"2024-04-16T07:49:53.198548+0000","last_peered":"2024-04-16T07:49:53.198548+0000","last_clean":"2024-04-16T07:49:53.198548+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:53.198548+0000","last_undegraded":"2024-04-16T07:49:53.198548+0000","last_fullsized":"2024-04-16T07:49:53.198548+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:49:21.822703+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:49:22.788948+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":231,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.852687+0000","last_change":"2024-04-16T07:49:22.853950+0000","last_active":"2024-04-16T07:49:22.852687+0000","last_peered":"2024-04-16T07:49:22.852687+0000","last_clean":"2024-04-16T07:48:22.913279+0000","last_became_active":"2024-04-16T07:49:22.847179+0000","last_became_peered":"2024-04-16T07:49:22.847179+0000","last_unstale":"2024-04-16T07:49:22.852687+0000","last_undegraded":"2024-04-16T07:49:22.852687+0000","last_fullsized":"2024-04-16T07:49:22.852687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"48'105","reported_seq":127,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.847019+0000","last_change":"2024-04-16T07:49:22.848219+0000","last_active":"2024-04-16T07:49:22.847019+0000","last_peered":"2024-04-16T07:49:22.847019+0000","last_clean":"2024-04-16T07:49:02.322013+0000","last_became_active":"2024-04-16T07:49:22.843784+0000","last_became_peered":"2024-04-16T07:49:22.843784+0000","last_unstale":"2024-04-16T07:49:22.847019+0000","last_undegraded":"2024-04-16T07:49:22.847019+0000","last_fullsized":"2024-04-16T07:49:22.847019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:49:22.790566+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":166,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.189582+0000","last_change":"2024-04-16T07:49:22.862015+0000","last_active":"2024-04-16T07:49:53.189582+0000","last_peered":"2024-04-16T07:49:53.189582+0000","last_clean":"2024-04-16T07:49:53.189582+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:53.189582+0000","last_undegraded":"2024-04-16T07:49:53.189582+0000","last_fullsized":"2024-04-16T07:49:53.189582+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":168,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:53.195326+0000","last_change":"2024-04-16T07:49:36.046350+0000","last_active":"2024-04-16T07:49:53.195326+0000","last_peered":"2024-04-16T07:49:53.195326+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:53.195326+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:53.195326+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":107},{"shards":"2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.801564+0000","last_change":"2024-04-16T07:48:47.921810+0000","last_active":"2024-04-16T07:49:22.801564+0000","last_peered":"2024-04-16T07:49:22.801564+0000","last_clean":"2024-04-16T07:49:22.801564+0000","last_became_active":"2024-04-16T07:48:47.913008+0000","last_became_peered":"2024-04-16T07:48:47.913008+0000","last_unstale":"2024-04-16T07:49:22.801564+0000","last_undegraded":"2024-04-16T07:49:22.801564+0000","last_fullsized":"2024-04-16T07:49:22.801564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.192178+0000","last_change":"2024-04-16T07:49:22.857045+0000","last_active":"2024-04-16T07:49:53.192178+0000","last_peered":"2024-04-16T07:49:53.192178+0000","last_clean":"2024-04-16T07:49:53.192178+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:53.192178+0000","last_undegraded":"2024-04-16T07:49:53.192178+0000","last_fullsized":"2024-04-16T07:49:53.192178+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:49:22.787540+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-16T07:49:22.864951+0000","last_change":"2024-04-16T07:49:22.866332+0000","last_active":"2024-04-16T07:49:22.864951+0000","last_peered":"2024-04-16T07:49:22.864951+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:49:22.858691+0000","last_became_peered":"2024-04-16T07:49:22.858691+0000","last_unstale":"2024-04-16T07:49:22.864951+0000","last_undegraded":"2024-04-16T07:49:22.864951+0000","last_fullsized":"2024-04-16T07:49:22.864951+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:49:21.774567+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:53.210443+0000","last_change":"2024-04-16T07:49:22.896203+0000","last_active":"2024-04-16T07:49:53.210443+0000","last_peered":"2024-04-16T07:49:53.210443+0000","last_clean":"2024-04-16T07:49:53.210443+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:53.210443+0000","last_undegraded":"2024-04-16T07:49:53.210443+0000","last_fullsized":"2024-04-16T07:49:53.210443+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":71303168,"num_objects":1088,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1088,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1814,"ondisk_log_size":1814,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32656,"kb_used_data":5320,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":5447680,"data_stored":79096164,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:02.878 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 69 16 1934 1918 1.7371 0 - 0.348198 2024-04-16T07:50:03.878 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 70 16 1934 1918 1.71228 0 - 0.348198 2024-04-16T07:50:04.195 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:50:04.878 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 71 16 1934 1918 1.68817 0 - 0.348198 2024-04-16T07:50:05.623 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:05.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:05.735+0000 7f24f8e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:05.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:05.743+0000 7f24f8e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:05.879 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 72 16 1935 1919 1.66559 0.00215517 29.1686 0.363217 2024-04-16T07:50:05.928 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:05.929 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:05.941 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-04-16T07:50:05.200888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145672,"kb_used_data":21928,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341688,"statfs":{"total":386547056640,"available":386397888512,"internally_reserved":0,"allocated":22454272,"data_stored":329437340,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7471648,"num_objects":-107,"num_object_clones":0,"num_object_copies":-321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-103,"num_read_kb":-91,"num_write":-242,"num_write_kb":-8614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.103635"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":163,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.209226+0000","last_change":"2024-04-16T07:50:04.269432+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:49:53.209226+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:53.209226+0000","last_undegraded":"2024-04-16T07:49:53.209226+0000","last_fullsized":"2024-04-16T07:49:53.209226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":206,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.200689+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:53.200689+0000","last_undegraded":"2024-04-16T07:49:53.200689+0000","last_fullsized":"2024-04-16T07:49:53.200689+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":3,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:45:33.069221+0000","last_change":"2024-04-16T07:50:04.275322+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:45:32.048077+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T07:45:33.069221+0000","last_undegraded":"2024-04-16T07:45:33.069221+0000","last_fullsized":"2024-04-16T07:45:33.069221+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":3,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:48:52.832531+0000","last_change":"2024-04-16T07:50:04.244280+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:48:51.768206+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T07:48:52.832531+0000","last_undegraded":"2024-04-16T07:48:52.832531+0000","last_fullsized":"2024-04-16T07:48:52.832531+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":167,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.190048+0000","last_change":"2024-04-16T07:50:04.231462+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:49:53.190048+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:53.190048+0000","last_undegraded":"2024-04-16T07:49:53.190048+0000","last_fullsized":"2024-04-16T07:49:53.190048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":169,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.197048+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:53.197048+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:53.197048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":119,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:35.823558+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:49:35.823558+0000","last_undegraded":"2024-04-16T07:49:35.823558+0000","last_fullsized":"2024-04-16T07:49:35.823558+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.194258+0000","last_change":"2024-04-16T07:50:04.241461+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:49:53.194258+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:53.194258+0000","last_undegraded":"2024-04-16T07:49:53.194258+0000","last_fullsized":"2024-04-16T07:49:53.194258+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:22.784306+0000","last_change":"2024-04-16T07:49:21.782628+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:48:22.899313+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:48:22.892718+0000","last_unstale":"2024-04-16T07:49:22.784306+0000","last_undegraded":"2024-04-16T07:49:22.784306+0000","last_fullsized":"2024-04-16T07:49:22.784306+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":136,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.211073+0000","last_change":"2024-04-16T07:50:04.256242+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:49:53.211073+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:53.211073+0000","last_undegraded":"2024-04-16T07:49:53.211073+0000","last_fullsized":"2024-04-16T07:49:53.211073+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182598,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32656,"kb_used_data":5320,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":5447680,"data_stored":79096164,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:05.942 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:06.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:06.051+0000 7f95d6b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:06.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:06.057+0000 7f95d6b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:06.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2024-04-16T07:50:06.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-16T07:50:06.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~27s 2024-04-16T07:50:06.308 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:06.308 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs inactive, 5 pgs peering","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 133/2949 objects degraded (4.510%), 6 pgs degraded","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":283,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":10},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149168128,"bytes_avail":386397888512,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":133,"degraded_total":2949,"degraded_ratio":0.045100033909799934},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:06.321 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:06.322 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:06.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:06.429+0000 7fdf1706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:06.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:06.437+0000 7fdf1706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:06.619 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:06.632 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-04-16T07:50:05.200888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145672,"kb_used_data":21928,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341688,"statfs":{"total":386547056640,"available":386397888512,"internally_reserved":0,"allocated":22454272,"data_stored":329437340,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7471648,"num_objects":-107,"num_object_clones":0,"num_object_copies":-321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-103,"num_read_kb":-91,"num_write":-242,"num_write_kb":-8614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.103635"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":163,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.209226+0000","last_change":"2024-04-16T07:50:04.269432+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:49:53.209226+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:53.209226+0000","last_undegraded":"2024-04-16T07:49:53.209226+0000","last_fullsized":"2024-04-16T07:49:53.209226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":206,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.200689+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:53.200689+0000","last_undegraded":"2024-04-16T07:49:53.200689+0000","last_fullsized":"2024-04-16T07:49:53.200689+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":3,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:45:33.069221+0000","last_change":"2024-04-16T07:50:04.275322+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:45:32.048077+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T07:45:33.069221+0000","last_undegraded":"2024-04-16T07:45:33.069221+0000","last_fullsized":"2024-04-16T07:45:33.069221+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":3,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:48:52.832531+0000","last_change":"2024-04-16T07:50:04.244280+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:48:51.768206+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T07:48:52.832531+0000","last_undegraded":"2024-04-16T07:48:52.832531+0000","last_fullsized":"2024-04-16T07:48:52.832531+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":167,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.190048+0000","last_change":"2024-04-16T07:50:04.231462+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:49:53.190048+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:53.190048+0000","last_undegraded":"2024-04-16T07:49:53.190048+0000","last_fullsized":"2024-04-16T07:49:53.190048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":169,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.197048+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:53.197048+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:53.197048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":119,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:35.823558+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:49:35.823558+0000","last_undegraded":"2024-04-16T07:49:35.823558+0000","last_fullsized":"2024-04-16T07:49:35.823558+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.194258+0000","last_change":"2024-04-16T07:50:04.241461+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:49:53.194258+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:53.194258+0000","last_undegraded":"2024-04-16T07:49:53.194258+0000","last_fullsized":"2024-04-16T07:49:53.194258+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:22.784306+0000","last_change":"2024-04-16T07:49:21.782628+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:48:22.899313+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:48:22.892718+0000","last_unstale":"2024-04-16T07:49:22.784306+0000","last_undegraded":"2024-04-16T07:49:22.784306+0000","last_fullsized":"2024-04-16T07:49:22.784306+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":136,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.211073+0000","last_change":"2024-04-16T07:50:04.256242+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:49:53.211073+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:53.211073+0000","last_undegraded":"2024-04-16T07:49:53.211073+0000","last_fullsized":"2024-04-16T07:49:53.211073+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182598,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32656,"kb_used_data":5320,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":5447680,"data_stored":79096164,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:06.879 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 73 16 1935 1919 1.64277 0 - 0.363217 2024-04-16T07:50:07.879 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 74 16 1935 1919 1.62057 0 - 0.363217 2024-04-16T07:50:08.879 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 75 16 1935 1919 1.59896 0 - 0.363217 2024-04-16T07:50:09.634 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:09.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:09.755+0000 7fb83782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:09.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:09.763+0000 7fb83782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:09.879 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 76 16 1935 1919 1.57793 0 - 0.363217 2024-04-16T07:50:09.947 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:09.948 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:09.960 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-04-16T07:50:09.100608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145672,"kb_used_data":21928,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341688,"statfs":{"total":386547056640,"available":386397888512,"internally_reserved":0,"allocated":22454272,"data_stored":329437340,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7471648,"num_objects":-107,"num_object_clones":0,"num_object_copies":-321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-103,"num_read_kb":-91,"num_write":-242,"num_write_kb":-8614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002296"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":163,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.209226+0000","last_change":"2024-04-16T07:50:04.269432+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:49:53.209226+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:53.209226+0000","last_undegraded":"2024-04-16T07:49:53.209226+0000","last_fullsized":"2024-04-16T07:49:53.209226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":206,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.200689+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:53.200689+0000","last_undegraded":"2024-04-16T07:49:53.200689+0000","last_fullsized":"2024-04-16T07:49:53.200689+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":3,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:45:33.069221+0000","last_change":"2024-04-16T07:50:04.275322+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:45:32.048077+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T07:45:33.069221+0000","last_undegraded":"2024-04-16T07:45:33.069221+0000","last_fullsized":"2024-04-16T07:45:33.069221+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":3,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:48:52.832531+0000","last_change":"2024-04-16T07:50:04.244280+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:48:51.768206+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T07:48:52.832531+0000","last_undegraded":"2024-04-16T07:48:52.832531+0000","last_fullsized":"2024-04-16T07:48:52.832531+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":167,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.190048+0000","last_change":"2024-04-16T07:50:04.231462+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:49:53.190048+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:53.190048+0000","last_undegraded":"2024-04-16T07:49:53.190048+0000","last_fullsized":"2024-04-16T07:49:53.190048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":169,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.197048+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:53.197048+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:53.197048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":119,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:35.823558+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:49:35.823558+0000","last_undegraded":"2024-04-16T07:49:35.823558+0000","last_fullsized":"2024-04-16T07:49:35.823558+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.194258+0000","last_change":"2024-04-16T07:50:04.241461+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:49:53.194258+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:53.194258+0000","last_undegraded":"2024-04-16T07:49:53.194258+0000","last_fullsized":"2024-04-16T07:49:53.194258+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:22.784306+0000","last_change":"2024-04-16T07:49:21.782628+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:48:22.899313+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:48:22.892718+0000","last_unstale":"2024-04-16T07:49:22.784306+0000","last_undegraded":"2024-04-16T07:49:22.784306+0000","last_fullsized":"2024-04-16T07:49:22.784306+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":136,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.211073+0000","last_change":"2024-04-16T07:50:04.256242+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:49:53.211073+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:53.211073+0000","last_undegraded":"2024-04-16T07:49:53.211073+0000","last_fullsized":"2024-04-16T07:49:53.211073+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182598,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32656,"kb_used_data":5320,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":5447680,"data_stored":79096164,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:09.961 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:10.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:10.069+0000 7fc28d39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:10.075+0000 7fc28d39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:10.326 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs inactive, 5 pgs peering","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 133/2949 objects degraded (4.510%), 6 pgs degraded","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":287,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":10},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149168128,"bytes_avail":386397888512,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":133,"degraded_total":2949,"degraded_ratio":0.045100033909799934},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:10.340 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:10.341 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:10.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:10.448+0000 7fd21f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:10.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:10.456+0000 7fd21f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:10.639 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:10.652 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-04-16T07:50:09.100608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145672,"kb_used_data":21928,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341688,"statfs":{"total":386547056640,"available":386397888512,"internally_reserved":0,"allocated":22454272,"data_stored":329437340,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7471648,"num_objects":-107,"num_object_clones":0,"num_object_copies":-321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-103,"num_read_kb":-91,"num_write":-242,"num_write_kb":-8614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002296"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":163,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.209226+0000","last_change":"2024-04-16T07:50:04.269432+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:49:53.209226+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:49:22.880458+0000","last_unstale":"2024-04-16T07:49:53.209226+0000","last_undegraded":"2024-04-16T07:49:53.209226+0000","last_fullsized":"2024-04-16T07:49:53.209226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":206,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.200689+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:49:53.200689+0000","last_undegraded":"2024-04-16T07:49:53.200689+0000","last_fullsized":"2024-04-16T07:49:53.200689+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":3,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:45:33.069221+0000","last_change":"2024-04-16T07:50:04.275322+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:45:32.048077+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T07:45:33.069221+0000","last_undegraded":"2024-04-16T07:45:33.069221+0000","last_fullsized":"2024-04-16T07:45:33.069221+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":3,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:48:52.832531+0000","last_change":"2024-04-16T07:50:04.244280+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:48:51.768206+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-16T07:48:52.832531+0000","last_undegraded":"2024-04-16T07:48:52.832531+0000","last_fullsized":"2024-04-16T07:48:52.832531+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":167,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.190048+0000","last_change":"2024-04-16T07:50:04.231462+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:49:53.190048+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:49:22.852725+0000","last_unstale":"2024-04-16T07:49:53.190048+0000","last_undegraded":"2024-04-16T07:49:53.190048+0000","last_fullsized":"2024-04-16T07:49:53.190048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":169,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.197048+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:49:53.197048+0000","last_undegraded":"2024-04-16T07:49:35.805005+0000","last_fullsized":"2024-04-16T07:49:53.197048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":119,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:35.823558+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:49:35.823558+0000","last_undegraded":"2024-04-16T07:49:35.823558+0000","last_fullsized":"2024-04-16T07:49:35.823558+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.194258+0000","last_change":"2024-04-16T07:50:04.241461+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:49:53.194258+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:49:22.847154+0000","last_unstale":"2024-04-16T07:49:53.194258+0000","last_undegraded":"2024-04-16T07:49:53.194258+0000","last_fullsized":"2024-04-16T07:49:53.194258+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:22.784306+0000","last_change":"2024-04-16T07:49:21.782628+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:48:22.899313+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:48:22.892718+0000","last_unstale":"2024-04-16T07:49:22.784306+0000","last_undegraded":"2024-04-16T07:49:22.784306+0000","last_fullsized":"2024-04-16T07:49:22.784306+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":136,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-16T07:49:53.211073+0000","last_change":"2024-04-16T07:50:04.256242+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:49:53.211073+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:49:22.885743+0000","last_unstale":"2024-04-16T07:49:53.211073+0000","last_undegraded":"2024-04-16T07:49:53.211073+0000","last_fullsized":"2024-04-16T07:49:53.211073+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182598,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32656,"kb_used_data":5320,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":5447680,"data_stored":79096164,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:10.879 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 77 16 1935 1919 1.55743 0 - 0.363217 2024-04-16T07:50:11.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2024-04-16T07:50:11.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-04-16T07:50:11.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~32s 2024-04-16T07:50:11.879 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 78 16 1935 1919 1.53747 0 - 0.363217 2024-04-16T07:50:12.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 79 16 1935 1919 1.518 0 - 0.363217 2024-04-16T07:50:13.653 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:13.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:13.775+0000 7f91f4e06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:13.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:13.782+0000 7f91f4e06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:13.879 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:50:13.881272+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:50:13.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:50:13.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 80 16 1935 1919 1.49903 0 - 0.363217 2024-04-16T07:50:13.966 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:13.966 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:13.978 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-16T07:50:13.101730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7471648,"num_objects":-107,"num_object_clones":0,"num_object_copies":-321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-103,"num_read_kb":-91,"num_write":-242,"num_write_kb":-8614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.908217"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:13.980 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:14.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:14.087+0000 7f953c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:14.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:14.094+0000 7f953c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:14.346 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:14.346 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs inactive, 2 pgs peering","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":291,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:14.359 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:14.360 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:14.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:14.467+0000 7f3377e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:14.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:14.474+0000 7f3377e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:14.657 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:14.658 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:14.669 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-16T07:50:13.101730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7471648,"num_objects":-107,"num_object_clones":0,"num_object_copies":-321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-103,"num_read_kb":-91,"num_write":-242,"num_write_kb":-8614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.908217"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:14.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 81 16 1935 1919 1.48052 0 - 0.363217 2024-04-16T07:50:15.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 82 16 1935 1919 1.46247 0 - 0.363217 2024-04-16T07:50:16.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 83 16 1935 1919 1.44485 0 - 0.363217 2024-04-16T07:50:16.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~38s 2024-04-16T07:50:16.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-04-16T07:50:16.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~38s 2024-04-16T07:50:17.672 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:17.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:17.793+0000 7f7a4442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:17.800+0000 7f7a4442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:17.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 84 16 1935 1919 1.42765 0 - 0.363217 2024-04-16T07:50:17.985 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:17.985 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:17.998 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-16T07:50:17.102825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.901937"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182608,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:17.999 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:18.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:18.107+0000 7f7b8986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:18.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:18.113+0000 7f7b8986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:18.366 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:18.367 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs inactive, 2 pgs peering","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":295,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:18.380 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:18.380 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:18.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:18.487+0000 7f1a9ea6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:18.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:18.495+0000 7f1a9ea6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:18.677 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:18.677 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:18.690 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-16T07:50:17.102825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.901937"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182608,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:18.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 85 16 1935 1919 1.41085 0 - 0.363217 2024-04-16T07:50:19.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 86 16 1935 1919 1.39445 0 - 0.363217 2024-04-16T07:50:20.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 87 16 1935 1919 1.37842 0 - 0.363217 2024-04-16T07:50:21.692 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:21.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:21.814+0000 7f041eead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:21.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:21.822+0000 7f041eead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:21.880 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 88 16 1935 1919 1.36276 0 - 0.363217 2024-04-16T07:50:22.004 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:22.004 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:22.016 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-16T07:50:21.103919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003311"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:22.018 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:22.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:22.126+0000 7fad89d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:22.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:22.132+0000 7fad89d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:22.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~43s 2024-04-16T07:50:22.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2024-04-16T07:50:22.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~43s 2024-04-16T07:50:22.382 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:22.382 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 3 pgs inactive, 2 pgs peering","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":299,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:22.395 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:22.396 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:22.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:22.503+0000 7f6c1d8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:22.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:22.510+0000 7f6c1d8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:22.693 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:22.694 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:22.706 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-16T07:50:21.103919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003311"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:22.881 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 89 16 1935 1919 1.34744 0 - 0.363217 2024-04-16T07:50:23.881 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 90 16 1935 1919 1.33247 0 - 0.363217 2024-04-16T07:50:24.881 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 91 16 1935 1919 1.31783 0 - 0.363217 2024-04-16T07:50:25.708 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:25.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:25.829+0000 7fcab039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:25.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:25.836+0000 7fcab039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:25.881 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 92 16 1935 1919 1.3035 0 - 0.363217 2024-04-16T07:50:26.020 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:26.021 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:26.033 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-04-16T07:50:25.105059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182618,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:26.034 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:26.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:26.142+0000 7f4e8b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:26.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:26.149+0000 7f4e8b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:26.400 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:26.400 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 5 pgs inactive, 2 pgs peering","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":303,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:26.414 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:26.414 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:26.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:26.521+0000 7f76b526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:26.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:26.529+0000 7f76b526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:26.714 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:26.715 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:26.727 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-04-16T07:50:25.105059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182618,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:26.881 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 93 16 1935 1919 1.28949 0 - 0.363217 2024-04-16T07:50:27.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2024-04-16T07:50:27.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~21s 2024-04-16T07:50:27.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~48s 2024-04-16T07:50:27.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 94 16 1935 1919 1.27577 0 - 0.363217 2024-04-16T07:50:28.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 95 16 1935 1919 1.26234 0 - 0.363217 2024-04-16T07:50:29.730 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:29.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:29.852+0000 7f820966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:29.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:29.860+0000 7f820966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:29.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 96 16 1935 1919 1.24919 0 - 0.363217 2024-04-16T07:50:30.052 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:30.052 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:30.065 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-04-16T07:50:29.106293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003467"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182618,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:30.067 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:30.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:30.178+0000 7f25d6f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:30.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:30.185+0000 7f25d6f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:30.454 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:30.455 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 5 pgs inactive, 2 pgs peering","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":307,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:30.468 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:30.469 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:30.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:30.578+0000 7f2f9342d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:30.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:30.586+0000 7f2f9342d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:30.777 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:30.777 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:30.790 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-04-16T07:50:29.106293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003467"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182618,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:30.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 97 16 1935 1919 1.23631 0 - 0.363217 2024-04-16T07:50:31.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 98 16 1935 1919 1.2237 0 - 0.363217 2024-04-16T07:50:32.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~54s 2024-04-16T07:50:32.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2024-04-16T07:50:32.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~54s 2024-04-16T07:50:32.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 99 16 1935 1919 1.21133 0 - 0.363217 2024-04-16T07:50:33.792 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:33.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:50:33.884024+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:50:33.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:50:33.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 100 16 1935 1919 1.19922 0 - 0.363217 2024-04-16T07:50:33.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:33.914+0000 7fb3ef46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:33.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:33.922+0000 7fb3ef46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:34.106 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:34.106 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:34.118 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-04-16T07:50:33.107583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003663"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182623,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:34.120 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:34.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:34.228+0000 7f748cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:34.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:34.234+0000 7f748cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:34.484 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 5 pgs inactive, 2 pgs peering","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":311,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:34.497 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:34.497 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:34.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:34.605+0000 7f0aecc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:34.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:34.613+0000 7f0aecc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:34.797 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:34.798 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:34.810 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-04-16T07:50:33.107583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003663"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182623,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:34.882 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 101 16 1935 1919 1.18735 0 - 0.363217 2024-04-16T07:50:35.883 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 102 16 1935 1919 1.17571 0 - 0.363217 2024-04-16T07:50:36.883 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 103 16 1935 1919 1.16429 0 - 0.363217 2024-04-16T07:50:37.812 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:37.883 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 104 16 1935 1919 1.1531 0 - 0.363217 2024-04-16T07:50:37.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:37.933+0000 7eff32e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:37.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:37.941+0000 7eff32e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:38.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~59s 2024-04-16T07:50:38.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~32s 2024-04-16T07:50:38.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~59s 2024-04-16T07:50:38.123 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:38.123 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:38.135 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-04-16T07:50:37.108669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003609"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:38.137 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:38.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:38.243+0000 7f018bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:38.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:38.250+0000 7f018bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:38.500 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:38.500 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 6 pgs inactive, 3 pgs peering","count":9},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":315,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:38.513 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:38.514 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:38.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:38.622+0000 7fb56e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:38.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:38.629+0000 7fb56e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:38.816 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:38.816 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:38.829 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-04-16T07:50:37.108669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003609"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:38.883 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 105 16 1935 1919 1.14212 0 - 0.363217 2024-04-16T07:50:39.883 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 106 16 1935 1919 1.13134 0 - 0.363217 2024-04-16T07:50:40.883 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 107 16 1935 1919 1.12077 0 - 0.363217 2024-04-16T07:50:41.830 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:41.883 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 108 16 1935 1919 1.11039 0 - 0.363217 2024-04-16T07:50:41.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:41.953+0000 7ff6a4a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:41.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:41.960+0000 7ff6a4a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:42.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:42.144 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:42.157 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-04-16T07:50:41.109673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003380"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:42.158 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:42.267+0000 7fe06b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:42.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:42.273+0000 7fe06b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:42.523 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:42.523 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 6 pgs inactive, 3 pgs peering","count":9},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":319,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:42.537 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:42.537 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:42.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:42.645+0000 7f846895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:42.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:42.652+0000 7f846895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:42.835 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:42.835 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:42.848 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-04-16T07:50:41.109673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003380"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:42.884 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 109 16 1935 1919 1.1002 0 - 0.363217 2024-04-16T07:50:43.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~64s 2024-04-16T07:50:43.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~37s 2024-04-16T07:50:43.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~64s 2024-04-16T07:50:43.884 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 110 16 1935 1919 1.0902 0 - 0.363217 2024-04-16T07:50:44.884 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 111 16 1935 1919 1.08038 0 - 0.363217 2024-04-16T07:50:45.850 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:45.884 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 112 16 1935 1919 1.07073 0 - 0.363217 2024-04-16T07:50:45.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:45.970+0000 7f08cca2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:45.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:45.978+0000 7f08cca2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:46.162 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:46.162 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:46.174 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-04-16T07:50:45.110727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003144"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:46.176 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:46.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:46.323+0000 7f5bd186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:46.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:46.329+0000 7f5bd186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:46.580 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:46.580 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 6 pgs inactive, 3 pgs peering","count":9},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":323,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:46.593 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:46.593 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:46.705 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:46.702+0000 7f9cefcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:46.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:46.709+0000 7f9cefcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:46.884 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 113 16 1935 1919 1.06126 0 - 0.363217 2024-04-16T07:50:46.899 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:46.899 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:46.911 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-04-16T07:50:45.110727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003144"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:47.884 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 114 16 1935 1919 1.05195 0 - 0.363217 2024-04-16T07:50:48.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~69s 2024-04-16T07:50:48.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~42s 2024-04-16T07:50:48.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~69s 2024-04-16T07:50:48.884 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 115 16 1935 1919 1.0428 0 - 0.363217 2024-04-16T07:50:49.884 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 116 16 1935 1919 1.03381 0 - 0.363217 2024-04-16T07:50:49.913 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:50.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:50.034+0000 7f2af559d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:50.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:50.042+0000 7f2af559d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:50.226 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:50.226 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:50.239 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-04-16T07:50:49.111837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003168"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:50.241 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:50.350 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:50.347+0000 7f93ed8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:50.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:50.353+0000 7f93ed8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:50.604 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:50.604 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 6 pgs inactive, 3 pgs peering","count":9},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":327,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:50.617 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:50.617 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:50.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:50.726+0000 7f6186c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:50.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:50.733+0000 7f6186c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:50.884 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 117 16 1935 1919 1.02498 0 - 0.363217 2024-04-16T07:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:50.920 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:50.933 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-04-16T07:50:49.111837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003168"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:51.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 118 16 1935 1919 1.01629 0 - 0.363217 2024-04-16T07:50:52.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 119 16 1935 1919 1.00775 0 - 0.363217 2024-04-16T07:50:53.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:50:53.886573+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:50:53.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:50:53.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 120 16 1935 1919 0.999351 0 - 0.363217 2024-04-16T07:50:53.935 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:54.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~75s 2024-04-16T07:50:54.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~48s 2024-04-16T07:50:54.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~75s 2024-04-16T07:50:54.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:54.055+0000 7f0e53f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:54.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:54.063+0000 7f0e53f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:54.245 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:54.246 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:54.258 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-04-16T07:50:53.113174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003501"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:54.259 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:54.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:54.367+0000 7fb8358ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:54.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:54.373+0000 7fb8358ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:54.626 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 6 pgs inactive, 3 pgs peering","count":9},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":331,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:54.641 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:54.641 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:54.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:54.747+0000 7f8e1706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:54.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:54.755+0000 7f8e1706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:54.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 121 16 1935 1919 0.991092 0 - 0.363217 2024-04-16T07:50:54.937 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:54.938 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:54.950 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-04-16T07:50:53.113174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003501"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:55.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 122 16 1935 1919 0.982968 0 - 0.363217 2024-04-16T07:50:56.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 123 16 1935 1919 0.974977 0 - 0.363217 2024-04-16T07:50:57.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 124 16 1935 1919 0.967114 0 - 0.363217 2024-04-16T07:50:57.952 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:58.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:58.072+0000 7fd1fd420640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:58.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:58.080+0000 7fd1fd420640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:58.263 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:58.263 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:58.276 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-04-16T07:50:57.114226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003498"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:58.277 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:50:58.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:58.385+0000 7ff2a1017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:58.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:58.391+0000 7ff2a1017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:58.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:58.644 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 6 pgs inactive, 5 pgs peering","count":11},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":335,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:50:58.658 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:50:58.658 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:50:58.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:58.766+0000 7f803d0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:58.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:50:58.773+0000 7f803d0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:50:58.885 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 125 16 1935 1919 0.959378 0 - 0.363217 2024-04-16T07:50:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:50:58.961 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:50:58.973 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-04-16T07:50:57.114226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003498"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:50:59.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2024-04-16T07:50:59.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~53s 2024-04-16T07:50:59.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~80s 2024-04-16T07:50:59.886 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 126 16 1935 1919 0.951764 0 - 0.363217 2024-04-16T07:51:00.886 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 127 16 1935 1919 0.944269 0 - 0.363217 2024-04-16T07:51:01.886 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 128 16 1935 1919 0.936892 0 - 0.363217 2024-04-16T07:51:01.976 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:02.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:02.097+0000 7ff4508ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:02.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:02.105+0000 7ff4508ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:02.290 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:02.291 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:02.303 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-04-16T07:51:01.115436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003599"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:02.304 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:02.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:02.411+0000 7fb885eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:02.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:02.418+0000 7fb885eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:02.671 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 6 pgs inactive, 5 pgs peering","count":11},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":339,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:02.684 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:02.684 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:02.758 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:02.758+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:02.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:02.792+0000 7f7af5f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:02.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:02.800+0000 7f7af5f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:02.886 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 129 16 1935 1919 0.929629 0 - 0.363217 2024-04-16T07:51:02.985 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:02.986 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:02.998 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-04-16T07:51:01.115436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003599"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:03.886 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 130 16 1935 1919 0.922478 0 - 0.363217 2024-04-16T07:51:04.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2024-04-16T07:51:04.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~58s 2024-04-16T07:51:04.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~85s 2024-04-16T07:51:04.886 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 131 16 1935 1919 0.915436 0 - 0.363217 2024-04-16T07:51:05.887 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 132 16 1935 1919 0.908501 0 - 0.363217 2024-04-16T07:51:06.001 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:06.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:06.121+0000 7f4e80e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:06.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:06.129+0000 7f4e80e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:06.313 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:06.313 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:06.326 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-04-16T07:51:05.116559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:06.328 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:06.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:06.436+0000 7f2909b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:06.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:06.442+0000 7f2909b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:06.693 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:06.693 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded","count":8},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 30 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":343,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:06.707 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:06.707 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:06.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:06.814+0000 7f2d2a22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:06.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:06.822+0000 7f2d2a22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:06.886 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 133 16 1935 1919 0.901671 0 - 0.363217 2024-04-16T07:51:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:07.007 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:07.020 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-04-16T07:51:05.116559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:07.759 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:07.759+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:07.887 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 134 16 1935 1919 0.894942 0 - 0.363217 2024-04-16T07:51:08.887 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 135 16 1935 1919 0.888313 0 - 0.363217 2024-04-16T07:51:09.887 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 136 16 1935 1919 0.881781 0 - 0.363217 2024-04-16T07:51:09.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~91s 2024-04-16T07:51:09.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~64s 2024-04-16T07:51:09.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~91s 2024-04-16T07:51:10.022 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:10.142+0000 7fc36b95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:10.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:10.150+0000 7fc36b95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:10.333 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:10.346 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-04-16T07:51:09.117823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003597"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:10.348 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:10.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:10.455+0000 7fea03a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:10.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:10.461+0000 7fea03a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:10.711 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 35 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":347,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:10.724 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:10.724 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:10.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:10.832+0000 7f0f90aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:10.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:10.840+0000 7f0f90aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:10.887 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 137 16 1935 1919 0.875345 0 - 0.363217 2024-04-16T07:51:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:11.024 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:11.036 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-04-16T07:51:09.117823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003597"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:11.887 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 138 16 1935 1919 0.869002 0 - 0.363217 2024-04-16T07:51:12.760 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:12.760+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:12.887 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 139 16 1935 1919 0.86275 0 - 0.363217 2024-04-16T07:51:13.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:51:13.889058+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:51:13.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:51:13.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 140 16 1935 1919 0.856587 0 - 0.363217 2024-04-16T07:51:14.039 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:14.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:14.159+0000 7f07ca824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:14.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:14.167+0000 7f07ca824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:14.351 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:14.351 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:14.363 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-04-16T07:51:13.119197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003760"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:14.365 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:14.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:14.473+0000 7ff3a7a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:14.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:14.480+0000 7ff3a7a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:14.733 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:14.733 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 40 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":352,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:14.746 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:14.746 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:14.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:14.854+0000 7fb46d664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:14.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:14.862+0000 7fb46d664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:14.887 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 141 16 1935 1919 0.850512 0 - 0.363217 2024-04-16T07:51:15.046 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:15.047 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:15.059 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-04-16T07:51:13.119197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003760"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:15.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~96s 2024-04-16T07:51:15.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~69s 2024-04-16T07:51:15.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~96s 2024-04-16T07:51:15.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 142 16 1935 1919 0.844523 0 - 0.363217 2024-04-16T07:51:16.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 143 16 1935 1919 0.838617 0 - 0.363217 2024-04-16T07:51:17.761 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:17.761+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:17.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 144 16 1935 1919 0.832793 0 - 0.363217 2024-04-16T07:51:18.061 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:18.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:18.181+0000 7fc85ea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:18.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:18.189+0000 7fc85ea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:18.372 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:18.372 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:18.385 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-04-16T07:51:17.120748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004188"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:18.386 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:18.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:18.494+0000 7f9726c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:18.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:18.500+0000 7f9726c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:18.751 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:18.751 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 40 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":356,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:18.764 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:18.764 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:18.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:18.871+0000 7fcb5122d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:18.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:18.879+0000 7fcb5122d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:18.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 145 16 1935 1919 0.82705 0 - 0.363217 2024-04-16T07:51:19.063 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:19.064 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:19.076 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-04-16T07:51:17.120748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004188"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:19.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 146 16 1935 1919 0.821385 0 - 0.363217 2024-04-16T07:51:20.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2024-04-16T07:51:20.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~74s 2024-04-16T07:51:20.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~101s 2024-04-16T07:51:20.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 147 16 1935 1919 0.815797 0 - 0.363217 2024-04-16T07:51:21.889 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 148 16 1935 1919 0.810285 0 - 0.363217 2024-04-16T07:51:22.078 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:22.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:22.199+0000 7f0ba539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:22.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:22.205+0000 7f0ba539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:22.388 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:22.388 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:22.400 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-04-16T07:51:21.121964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004141"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:22.402 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:22.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:22.511+0000 7f302e460640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:22.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:22.516+0000 7f302e460640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:22.761 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:22.761+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:22.768 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:22.769 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 45 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":360,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:22.782 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:22.782 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:22.888 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 149 16 1935 1919 0.804847 0 - 0.363217 2024-04-16T07:51:22.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:22.891+0000 7f56fdaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:22.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:22.898+0000 7f56fdaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:23.082 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:23.082 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:23.094 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-04-16T07:51:21.121964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004141"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:23.889 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 150 16 1935 1919 0.799482 0 - 0.363217 2024-04-16T07:51:24.889 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 151 16 1935 1919 0.794187 0 - 0.363217 2024-04-16T07:51:25.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~107s 2024-04-16T07:51:25.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~80s 2024-04-16T07:51:25.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~107s 2024-04-16T07:51:25.889 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 152 16 1935 1919 0.788962 0 - 0.363217 2024-04-16T07:51:26.096 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:26.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:26.216+0000 7f414e42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:26.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:26.224+0000 7f414e42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:26.408 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:26.408 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:26.420 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-04-16T07:51:25.123333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004135"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182678,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:26.422 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:26.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:26.530+0000 7f6d0ab5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:26.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:26.537+0000 7f6d0ab5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:26.790 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:26.790 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 50 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":364,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:26.803 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:26.803 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:26.889 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 153 16 1935 1919 0.783806 0 - 0.363217 2024-04-16T07:51:26.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:26.911+0000 7fa76406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:26.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:26.918+0000 7fa76406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:27.103 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:27.104 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:27.116 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-04-16T07:51:25.123333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004135"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182678,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:27.762 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:27.762+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:27.889 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 154 16 1935 1919 0.778716 0 - 0.363217 2024-04-16T07:51:28.889 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 155 16 1935 1919 0.773692 0 - 0.363217 2024-04-16T07:51:29.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 156 16 1935 1919 0.768733 0 - 0.363217 2024-04-16T07:51:30.120 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:30.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:30.240+0000 7f1b0706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:30.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:30.247+0000 7f1b0706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:30.431 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:30.432 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:30.443 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-04-16T07:51:29.124570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003822"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182678,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:30.444 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:30.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:30.552+0000 7fbaea4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:30.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:30.558+0000 7fbaea4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:30.809 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:30.809 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 55 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":368,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:30.822 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:30.822 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:30.889 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 157 16 1935 1919 0.763836 0 - 0.363217 2024-04-16T07:51:30.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:30.928+0000 7fcd5d064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:30.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:30.936+0000 7fcd5d064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:31.120 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:31.121 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:31.133 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-04-16T07:51:29.124570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003822"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182678,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:31.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~112s 2024-04-16T07:51:31.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~85s 2024-04-16T07:51:31.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~112s 2024-04-16T07:51:31.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 158 16 1935 1919 0.759002 0 - 0.363217 2024-04-16T07:51:32.763 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:32.763+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:32.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 159 16 1935 1919 0.754228 0 - 0.363217 2024-04-16T07:51:33.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:51:33.891509+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:51:33.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:51:33.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 160 16 1935 1919 0.749514 0 - 0.363217 2024-04-16T07:51:34.135 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:34.256+0000 7faef455d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:34.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:34.264+0000 7faef455d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:34.448 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:34.460 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-04-16T07:51:33.126153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004188"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:34.461 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:34.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:34.569+0000 7f4b5bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:34.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:34.576+0000 7f4b5bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:34.829 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:34.829 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 60 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":372,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:34.843 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:34.843 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:34.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 161 16 1935 1919 0.744859 0 - 0.363217 2024-04-16T07:51:34.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:34.950+0000 7f0fb5ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:34.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:34.957+0000 7f0fb5ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:35.142 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:35.142 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:35.155 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-04-16T07:51:35.126921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004290"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:35.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 162 16 1935 1919 0.740261 0 - 0.363217 2024-04-16T07:51:36.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~117s 2024-04-16T07:51:36.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~90s 2024-04-16T07:51:36.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~117s 2024-04-16T07:51:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T07:51:36.796 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:51:36.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 163 16 1935 1919 0.735719 0 - 0.363217 2024-04-16T07:51:37.763 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:37.763+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:37.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 164 16 1935 1919 0.731233 0 - 0.363217 2024-04-16T07:51:38.156 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:38.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:38.279+0000 7f013ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:38.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:38.287+0000 7f013ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:38.469 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:38.470 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:38.481 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-04-16T07:51:37.127646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004312"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:38.483 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:38.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:38.590+0000 7fb5160ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:38.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:38.597+0000 7fb5160ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:38.797 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:51:38.854 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:38.854 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 60 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":376,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:38.868 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:38.868 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:38.890 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 165 16 1935 1919 0.726802 0 - 0.363217 2024-04-16T07:51:38.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:38.909+0000 7fce0386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:38.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:38.914+0000 7fce0386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:38.919 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:51:38.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:38.977+0000 7f80958ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:38.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:38.985+0000 7f80958ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:39.173 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:39.174 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:39.186 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-04-16T07:51:39.128405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004406"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:39.891 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 166 16 1935 1919 0.722424 0 - 0.363217 2024-04-16T07:51:40.891 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 167 16 1935 1919 0.718098 0 - 0.363217 2024-04-16T07:51:41.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~122s 2024-04-16T07:51:41.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~95s 2024-04-16T07:51:41.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~122s 2024-04-16T07:51:41.891 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 168 16 1935 1919 0.713823 0 - 0.363217 2024-04-16T07:51:42.189 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:42.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:42.310+0000 7f865119d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:42.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:42.317+0000 7f865119d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:42.500 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:42.500 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:42.513 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-04-16T07:51:41.129045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004474"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:42.514 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:42.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:42.622+0000 7f369419d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:42.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:42.628+0000 7f369419d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:42.764 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:42.764+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:42.880 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:42.880 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 65 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":380,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:42.891 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 169 16 1935 1919 0.7096 0 - 0.363217 2024-04-16T07:51:42.893 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:42.894 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:43.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:43.001+0000 7f2949eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:43.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:43.008+0000 7f2949eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:43.192 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:43.192 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:43.204 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-04-16T07:51:43.129680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004269"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:43.891 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 170 16 1935 1919 0.705426 0 - 0.363217 2024-04-16T07:51:44.891 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 171 16 1935 1919 0.7013 0 - 0.363217 2024-04-16T07:51:45.891 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 172 16 1935 1919 0.697223 0 - 0.363217 2024-04-16T07:51:46.206 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:46.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:46.326+0000 7fb2d262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:46.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:46.334+0000 7fb2d262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:46.517 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:46.518 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:46.530 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-04-16T07:51:45.130305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004152"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:46.532 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:46.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:46.638+0000 7f512f4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:46.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:46.645+0000 7f512f4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:46.891 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 173 16 1935 1919 0.693193 0 - 0.363217 2024-04-16T07:51:46.900 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:46.901 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 70 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":384,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:46.914 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:46.914 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:47.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:47.022+0000 7f2a7442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:47.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:47.030+0000 7f2a7442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:47.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~128s 2024-04-16T07:51:47.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~101s 2024-04-16T07:51:47.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~128s 2024-04-16T07:51:47.212 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:47.213 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:47.225 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-04-16T07:51:47.131109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004187"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:47.764 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:47.765+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:47.891 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 174 16 1935 1919 0.689209 0 - 0.363217 2024-04-16T07:51:48.892 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 175 16 1935 1919 0.685271 0 - 0.363217 2024-04-16T07:51:49.892 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 176 16 1935 1919 0.681377 0 - 0.363217 2024-04-16T07:51:50.227 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:50.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:50.348+0000 7f00ad55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:50.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:50.356+0000 7f00ad55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:50.554 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-04-16T07:51:49.131723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004077"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:50.556 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:50.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:50.663+0000 7fa0eb95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:50.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:50.669+0000 7fa0eb95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:50.892 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 177 16 1935 1919 0.677528 0 - 0.363217 2024-04-16T07:51:50.923 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:50.923 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 75 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":388,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:50.936 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:50.937 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:51.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:51.044+0000 7f7f08c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:51.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:51.052+0000 7f7f08c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:51.238 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:51.239 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:51.251 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-04-16T07:51:51.132459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004054"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:51.892 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 178 16 1935 1919 0.673721 0 - 0.363217 2024-04-16T07:51:52.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~133s 2024-04-16T07:51:52.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~106s 2024-04-16T07:51:52.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~133s 2024-04-16T07:51:52.765 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:52.765+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:52.892 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 179 16 1935 1919 0.669957 0 - 0.363217 2024-04-16T07:51:53.892 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:51:53.893865+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:51:53.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:51:53.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 180 16 1935 1919 0.666236 0 - 0.363217 2024-04-16T07:51:54.254 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:54.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:54.374+0000 7f514de24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:54.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:54.382+0000 7f514de24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:54.566 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:54.567 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:54.579 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-04-16T07:51:53.133095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004050"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:54.581 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:54.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:54.688+0000 7f3723e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:54.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:54.694+0000 7f3723e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:54.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 181 16 1935 1919 0.662555 0 - 0.363217 2024-04-16T07:51:54.948 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:54.948 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 80 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":392,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:54.961 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:54.962 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:55.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:55.069+0000 7f11eb424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:55.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:55.077+0000 7f11eb424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:55.265 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:55.265 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:55.278 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-04-16T07:51:55.133650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003969"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:55.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 182 16 1935 1919 0.658914 0 - 0.363217 2024-04-16T07:51:56.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 183 16 1935 1919 0.655314 0 - 0.363217 2024-04-16T07:51:57.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2024-04-16T07:51:57.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~111s 2024-04-16T07:51:57.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~138s 2024-04-16T07:51:57.766 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:51:57.766+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:51:57.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 184 16 1935 1919 0.651752 0 - 0.363217 2024-04-16T07:51:58.280 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:58.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:58.400+0000 7f57ccbfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:58.408+0000 7f57ccbfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:58.591 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:58.592 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:58.604 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-04-16T07:51:57.134242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003936"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:58.605 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:51:58.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:58.713+0000 7f4ddb0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:58.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:58.719+0000 7f4ddb0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:58.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 185 16 1935 1919 0.648229 0 - 0.363217 2024-04-16T07:51:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:58.974 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 80 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":396,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:51:58.987 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:51:58.987 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:51:59.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:59.096+0000 7f7b26d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:59.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:51:59.103+0000 7f7b26d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:51:59.285 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:51:59.286 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:51:59.298 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-04-16T07:51:59.134844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003735"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:51:59.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 186 16 1935 1919 0.644744 0 - 0.363217 2024-04-16T07:52:00.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 187 16 1935 1919 0.641296 0 - 0.363217 2024-04-16T07:52:01.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 188 16 1935 1919 0.637885 0 - 0.363217 2024-04-16T07:52:02.301 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:02.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:02.422+0000 7f0851310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:02.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:02.430+0000 7f0851310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:02.613 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:02.614 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:02.626 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-04-16T07:52:01.135578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003854"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:02.627 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:02.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:02.735+0000 7fd25f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:02.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:02.742+0000 7fd25f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:02.766 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:02.767+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:02.893 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 189 16 1935 1919 0.63451 0 - 0.363217 2024-04-16T07:52:02.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~144s 2024-04-16T07:52:02.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~117s 2024-04-16T07:52:02.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~144s 2024-04-16T07:52:02.995 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:02.995 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 85 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":400,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:03.008 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:03.008 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:03.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:03.115+0000 7ff500ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:03.123+0000 7ff500ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:03.310 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:03.310 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:03.323 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-04-16T07:52:03.136204+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003745"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:03.894 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 190 16 1935 1919 0.631171 0 - 0.363217 2024-04-16T07:52:04.894 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 191 16 1935 1919 0.627866 0 - 0.363217 2024-04-16T07:52:05.894 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 192 16 1935 1919 0.624596 0 - 0.363217 2024-04-16T07:52:06.325 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:06.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:06.445+0000 7f173946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:06.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:06.453+0000 7f173946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:06.634 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:06.635 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:06.647 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-04-16T07:52:05.136775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003679"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:06.649 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:06.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:06.756+0000 7f227ab5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:06.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:06.763+0000 7f227ab5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:06.894 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 193 16 1935 1919 0.62136 0 - 0.363217 2024-04-16T07:52:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:07.016 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 90 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":404,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:07.029 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:07.030 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:07.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:07.136+0000 7f74486ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:07.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:07.143+0000 7f74486ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:07.326 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:07.327 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:07.339 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-04-16T07:52:07.137526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003876"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:07.767 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:07.767+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:07.894 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 194 16 1935 1919 0.618157 0 - 0.363217 2024-04-16T07:52:08.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~149s 2024-04-16T07:52:08.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~122s 2024-04-16T07:52:08.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~149s 2024-04-16T07:52:08.894 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 195 16 1935 1919 0.614987 0 - 0.363217 2024-04-16T07:52:09.894 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 196 16 1935 1919 0.611849 0 - 0.363217 2024-04-16T07:52:10.341 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:10.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:10.463+0000 7f949875d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:10.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:10.471+0000 7f949875d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:10.654 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:10.666 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-04-16T07:52:09.138291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004049"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:10.667 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:10.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:10.775+0000 7f949c620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:10.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:10.781+0000 7f949c620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:10.894 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 197 16 1935 1919 0.608743 0 - 0.363217 2024-04-16T07:52:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:11.034 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 95 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":408,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:11.047 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:11.047 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:11.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:11.155+0000 7f4c672ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:11.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:11.162+0000 7f4c672ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:11.349 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:11.361 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-04-16T07:52:11.139037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004193"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:11.894 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 198 16 1935 1919 0.605669 0 - 0.363217 2024-04-16T07:52:12.768 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:12.768+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:12.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 199 16 1935 1919 0.602625 0 - 0.363217 2024-04-16T07:52:13.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~154s 2024-04-16T07:52:13.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~127s 2024-04-16T07:52:13.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~154s 2024-04-16T07:52:13.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:52:13.896347+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:52:13.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:52:13.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 200 16 1935 1919 0.599612 0 - 0.363217 2024-04-16T07:52:14.364 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:14.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:14.485+0000 7f3ba359d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:14.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:14.493+0000 7f3ba359d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:14.674 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:14.675 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:14.687 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-04-16T07:52:13.139754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004176"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:14.689 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:14.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:14.797+0000 7f66c84ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:14.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:14.803+0000 7f66c84ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:14.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 201 16 1935 1919 0.596629 0 - 0.363217 2024-04-16T07:52:15.055 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:15.056 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 100 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":412,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:15.069 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:15.069 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:15.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:15.176+0000 7f7e61b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:15.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:15.184+0000 7f7e61b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:15.368 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:15.368 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:15.380 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-04-16T07:52:15.140478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004274"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:15.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 202 16 1935 1919 0.593675 0 - 0.363217 2024-04-16T07:52:16.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 203 16 1935 1919 0.590751 0 - 0.363217 2024-04-16T07:52:17.769 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:17.769+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:17.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 204 16 1935 1919 0.587855 0 - 0.363217 2024-04-16T07:52:18.382 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:18.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:18.503+0000 7f10d3ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:18.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:18.510+0000 7f10d3ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:18.695 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:18.707 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-04-16T07:52:17.141141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004365"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:18.708 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:18.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:18.816+0000 7fe61615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:18.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:18.823+0000 7fe61615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:18.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~160s 2024-04-16T07:52:18.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~133s 2024-04-16T07:52:18.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~160s 2024-04-16T07:52:18.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 205 16 1935 1919 0.584987 0 - 0.363217 2024-04-16T07:52:19.075 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:19.075 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 100 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":416,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:19.088 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:19.088 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:19.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:19.196+0000 7fc1d94ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:19.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:19.203+0000 7fc1d94ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:19.385 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:19.386 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:19.398 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-04-16T07:52:19.141759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004233"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:19.895 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 206 16 1935 1919 0.582148 0 - 0.363217 2024-04-16T07:52:20.896 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 207 16 1935 1919 0.579336 0 - 0.363217 2024-04-16T07:52:21.896 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 208 16 1935 1919 0.57655 0 - 0.363217 2024-04-16T07:52:22.400 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:22.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:22.522+0000 7faaf815d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:22.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:22.530+0000 7faaf815d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:22.715 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:22.715 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:22.727 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-16T07:52:21.142544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004252"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:22.729 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:22.769 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:22.769+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:22.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:22.837+0000 7f1f148ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:22.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:22.844+0000 7f1f148ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:22.896 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 209 16 1935 1919 0.573792 0 - 0.363217 2024-04-16T07:52:23.097 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:23.097 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 105 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":420,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:23.110 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:23.111 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:23.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:23.259+0000 7f7d72a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:23.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:23.266+0000 7f7d72a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:23.449 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:23.450 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:23.462 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-04-16T07:52:23.143384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004346"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:23.896 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 210 16 1935 1919 0.571059 0 - 0.363217 2024-04-16T07:52:24.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~165s 2024-04-16T07:52:24.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~138s 2024-04-16T07:52:24.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~165s 2024-04-16T07:52:24.896 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 211 16 1935 1919 0.568353 0 - 0.363217 2024-04-16T07:52:25.896 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 212 16 1935 1919 0.565672 0 - 0.363217 2024-04-16T07:52:26.465 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:26.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:26.585+0000 7fba9b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:26.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:26.593+0000 7fba9b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:26.777 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:26.778 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:26.790 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-04-16T07:52:25.144022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004267"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:26.791 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:26.896 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 213 16 1935 1919 0.563016 0 - 0.363217 2024-04-16T07:52:26.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:26.899+0000 7f33a646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:26.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:26.906+0000 7f33a646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:27.162 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:27.163 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 110 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":424,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:27.176 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:27.176 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:27.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:27.284+0000 7f05478ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:27.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:27.291+0000 7f05478ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:27.475 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:27.476 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:27.488 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-04-16T07:52:27.144772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004293"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:27.770 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:27.770+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:27.897 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 214 16 1935 1919 0.560385 0 - 0.363217 2024-04-16T07:52:28.896 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 215 16 1935 1919 0.557779 0 - 0.363217 2024-04-16T07:52:29.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2024-04-16T07:52:29.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~143s 2024-04-16T07:52:29.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~170s 2024-04-16T07:52:29.897 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 216 16 1935 1919 0.555197 0 - 0.363217 2024-04-16T07:52:30.490 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:30.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:30.610+0000 7f768ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:30.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:30.618+0000 7f768ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:30.803 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:30.804 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:30.816 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-04-16T07:52:29.145377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004236"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:30.817 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:30.897 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 217 16 1935 1919 0.552638 0 - 0.363217 2024-04-16T07:52:30.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:30.926+0000 7f369c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:30.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:30.932+0000 7f369c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:31.183 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:31.183 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 115 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":428,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:31.196 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:31.196 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:31.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:31.304+0000 7f2153d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:31.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:31.312+0000 7f2153d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:31.497 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:31.509 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-04-16T07:52:31.146075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004315"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:31.897 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 218 16 1935 1919 0.550103 0 - 0.363217 2024-04-16T07:52:32.771 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:32.771+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:32.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 219 16 1935 1919 0.547591 0 - 0.363217 2024-04-16T07:52:33.897 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:52:33.898652+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:52:33.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:52:33.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 220 16 1935 1919 0.545102 0 - 0.363217 2024-04-16T07:52:34.511 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:34.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:34.634+0000 7f2db7f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:34.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:34.642+0000 7f2db7f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:34.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~176s 2024-04-16T07:52:34.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~148s 2024-04-16T07:52:34.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~176s 2024-04-16T07:52:34.828 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:34.829 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:34.841 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-04-16T07:52:33.146718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004174"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:34.842 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:34.897 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 221 16 1935 1919 0.542636 0 - 0.363217 2024-04-16T07:52:34.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:34.949+0000 7fd44f82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:34.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:34.956+0000 7fd44f82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:35.210 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:35.210 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 120 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":432,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:35.223 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:35.224 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:35.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:35.332+0000 7fd3ade6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:35.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:35.339+0000 7fd3ade6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:35.524 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:35.524 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:35.537 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-04-16T07:52:35.147437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004053"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:35.897 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 222 16 1935 1919 0.540192 0 - 0.363217 2024-04-16T07:52:36.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 223 16 1935 1919 0.537769 0 - 0.363217 2024-04-16T07:52:37.772 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:37.772+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:37.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 224 16 1935 1919 0.535368 0 - 0.363217 2024-04-16T07:52:38.539 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:38.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:38.660+0000 7f3606817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:38.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:38.668+0000 7f3606817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:38.856 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:38.857 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:38.868 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-04-16T07:52:37.148117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004095"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:38.870 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:38.897 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 225 16 1935 1919 0.532989 0 - 0.363217 2024-04-16T07:52:38.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:38.977+0000 7f2acfc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:38.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:38.984+0000 7f2acfc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:39.236 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:39.236 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 125 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":436,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:39.250 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:39.250 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:39.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:39.397+0000 7fc3ac8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:39.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:39.404+0000 7fc3ac8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:39.587 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:39.599 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-04-16T07:52:39.148723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003950"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:39.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 226 16 1935 1919 0.530631 0 - 0.363217 2024-04-16T07:52:40.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~181s 2024-04-16T07:52:40.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~154s 2024-04-16T07:52:40.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~181s 2024-04-16T07:52:40.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 227 16 1935 1919 0.528293 0 - 0.363217 2024-04-16T07:52:41.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 228 16 1935 1919 0.525976 0 - 0.363217 2024-04-16T07:52:42.601 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:42.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:42.721+0000 7fa72902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:42.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:42.729+0000 7fa72902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:42.772 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:42.772+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:42.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 229 16 1935 1919 0.523679 0 - 0.363217 2024-04-16T07:52:42.916 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:42.916 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:42.929 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-04-16T07:52:41.149348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003971"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:42.930 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:43.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:43.038+0000 7f9da579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:43.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:43.044+0000 7f9da579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:43.298 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:43.298 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 130 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":440,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:43.312 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:43.312 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:43.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:43.419+0000 7f656362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:43.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:43.426+0000 7f656362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:43.613 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:43.613 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:43.625 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-04-16T07:52:43.150004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003928"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:43.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 230 16 1935 1919 0.521402 0 - 0.363217 2024-04-16T07:52:44.898 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 231 16 1935 1919 0.519145 0 - 0.363217 2024-04-16T07:52:45.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~186s 2024-04-16T07:52:45.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~159s 2024-04-16T07:52:45.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~186s 2024-04-16T07:52:45.899 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 232 16 1935 1919 0.516908 0 - 0.363217 2024-04-16T07:52:46.627 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:46.749+0000 7f1d330ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:46.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:46.756+0000 7f1d330ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:46.899 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 233 16 1935 1919 0.514689 0 - 0.363217 2024-04-16T07:52:46.941 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:46.941 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:46.954 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-04-16T07:52:45.150655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003936"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:46.956 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:47.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:47.064+0000 7fb719e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:47.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:47.069+0000 7fb719e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:47.327 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:47.327 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 130 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":444,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:47.340 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:47.340 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:47.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:47.448+0000 7fe1344ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:47.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:47.454+0000 7fe1344ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:47.638 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:47.639 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:47.651 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-04-16T07:52:47.151498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004061"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:47.773 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:47.774+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:47.899 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 234 16 1935 1919 0.51249 0 - 0.363217 2024-04-16T07:52:48.899 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 235 16 1935 1919 0.510309 0 - 0.363217 2024-04-16T07:52:49.899 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 236 16 1935 1919 0.508147 0 - 0.363217 2024-04-16T07:52:50.653 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:50.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~191s 2024-04-16T07:52:50.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~164s 2024-04-16T07:52:50.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~191s 2024-04-16T07:52:50.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:50.774+0000 7fc9b13e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:50.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:50.782+0000 7fc9b13e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:50.899 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 237 16 1935 1919 0.506003 0 - 0.363217 2024-04-16T07:52:50.964 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:50.964 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:50.976 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-04-16T07:52:49.152351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004234"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:50.978 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:51.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:51.086+0000 7fcee70ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:51.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:51.092+0000 7fcee70ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:51.346 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:51.346 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 135 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":448,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:51.359 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:51.359 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:51.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:51.467+0000 7fd4d226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:51.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:51.474+0000 7fd4d226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:51.657 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:51.658 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:51.670 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-04-16T07:52:51.152968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004245"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:51.899 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 238 16 1935 1919 0.503876 0 - 0.363217 2024-04-16T07:52:52.774 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:52.774+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:52.899 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 239 16 1935 1919 0.501768 0 - 0.363217 2024-04-16T07:52:53.900 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:52:53.901070+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:52:53.900 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:52:53.900 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 240 16 1935 1919 0.499677 0 - 0.363217 2024-04-16T07:52:54.672 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:54.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:54.794+0000 7f4e04f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:54.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:54.801+0000 7f4e04f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:54.900 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 241 16 1935 1919 0.497604 0 - 0.363217 2024-04-16T07:52:54.983 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:54.984 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:54.996 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-04-16T07:52:53.153660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004311"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:54.997 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:55.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:55.106+0000 7fbe584ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:55.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:55.112+0000 7fbe584ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:55.363 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:55.363 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 140 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":452,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:55.377 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:55.377 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:55.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:55.484+0000 7f5931413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:55.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:55.491+0000 7f5931413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:55.673 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:55.674 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:55.686 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-04-16T07:52:55.154324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004320"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:55.900 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 242 16 1935 1919 0.495548 0 - 0.363217 2024-04-16T07:52:56.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~197s 2024-04-16T07:52:56.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~170s 2024-04-16T07:52:56.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~197s 2024-04-16T07:52:56.900 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 243 16 1935 1919 0.493508 0 - 0.363217 2024-04-16T07:52:57.774 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:52:57.774+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:52:57.900 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 244 16 1935 1919 0.491486 0 - 0.363217 2024-04-16T07:52:58.687 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:58.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:58.810+0000 7fdb66f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:58.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:58.818+0000 7fdb66f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:58.900 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 245 16 1935 1919 0.48948 0 - 0.363217 2024-04-16T07:52:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:59.003 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:59.015 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-04-16T07:52:57.155036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004381"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:59.017 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:52:59.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:59.123+0000 7f8321cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:59.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:59.130+0000 7f8321cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:59.381 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:59.381 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 140 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":456,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:52:59.395 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:52:59.395 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:52:59.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:59.503+0000 7f0e51ffa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:59.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:52:59.510+0000 7f0e51ffa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:52:59.694 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:52:59.694 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:52:59.707 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-04-16T07:52:59.155717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004218"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:52:59.900 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 246 16 1935 1919 0.48749 0 - 0.363217 2024-04-16T07:53:00.901 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 247 16 1935 1919 0.485516 0 - 0.363217 2024-04-16T07:53:01.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~202s 2024-04-16T07:53:01.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~175s 2024-04-16T07:53:01.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~202s 2024-04-16T07:53:01.901 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 248 16 1935 1919 0.483559 0 - 0.363217 2024-04-16T07:53:02.709 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:02.775 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:02.775+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:02.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:02.831+0000 7f933aa13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:02.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:02.838+0000 7f933aa13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:02.901 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 249 16 1935 1919 0.481617 0 - 0.363217 2024-04-16T07:53:03.020 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:03.020 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:03.033 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-04-16T07:53:01.156437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004085"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:03.034 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:03.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:03.142+0000 7fa8e1a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:03.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:03.148+0000 7fa8e1a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:03.401 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:03.402 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 145 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":460,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:03.415 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:03.415 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:03.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:03.523+0000 7f2ac462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:03.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:03.530+0000 7f2ac462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:03.713 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:03.714 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:03.726 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-04-16T07:53:03.157173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004204"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:03.901 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 250 16 1935 1919 0.47969 0 - 0.363217 2024-04-16T07:53:04.901 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 251 16 1935 1919 0.477779 0 - 0.363217 2024-04-16T07:53:05.901 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 252 16 1935 1919 0.475883 0 - 0.363217 2024-04-16T07:53:06.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~207s 2024-04-16T07:53:06.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~180s 2024-04-16T07:53:06.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~207s 2024-04-16T07:53:06.728 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:06.849+0000 7f4393c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:06.857+0000 7f4393c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:06.901 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 253 16 1935 1919 0.474002 0 - 0.363217 2024-04-16T07:53:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:07.040 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:07.052 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-04-16T07:53:05.157792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004132"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:07.054 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:07.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:07.162+0000 7f784575d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:07.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:07.169+0000 7f784575d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:07.420 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:07.420 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 150 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":464,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:07.433 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:07.433 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:07.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:07.541+0000 7f83b8ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:07.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:07.549+0000 7f83b8ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:07.731 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:07.731 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:07.744 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-04-16T07:53:07.158527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004202"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:07.775 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:07.776+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:07.901 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 254 16 1935 1919 0.472136 0 - 0.363217 2024-04-16T07:53:08.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 255 16 1935 1919 0.470285 0 - 0.363217 2024-04-16T07:53:09.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 256 16 1935 1919 0.468447 0 - 0.363217 2024-04-16T07:53:10.745 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:10.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:10.867+0000 7f43b0d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:10.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:10.874+0000 7f43b0d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:10.901 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 257 16 1935 1919 0.466625 0 - 0.363217 2024-04-16T07:53:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:11.060 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:11.072 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-04-16T07:53:09.159266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004230"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:11.074 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:11.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:11.181+0000 7fcee2cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:11.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:11.188+0000 7fcee2cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:11.441 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 155 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":468,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:11.454 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:11.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:11.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:11.562+0000 7f232479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:11.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:11.570+0000 7f232479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:11.752 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:11.752 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:11.764 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-04-16T07:53:11.159987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004270"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:11.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 258 16 1935 1919 0.464816 0 - 0.363217 2024-04-16T07:53:11.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~213s 2024-04-16T07:53:11.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~186s 2024-04-16T07:53:11.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~213s 2024-04-16T07:53:12.777 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:12.777+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:12.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 259 16 1935 1919 0.463022 0 - 0.363217 2024-04-16T07:53:13.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:53:13.903503+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:53:13.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:53:13.903 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 260 16 1935 1919 0.461241 0 - 0.363217 2024-04-16T07:53:14.767 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:14.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:14.888+0000 7eff3506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:14.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:14.895+0000 7eff3506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:14.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 261 16 1935 1919 0.459474 0 - 0.363217 2024-04-16T07:53:15.081 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:15.081 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:15.094 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-04-16T07:53:13.160650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004213"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:15.095 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:15.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:15.203+0000 7fd169e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:15.209+0000 7fd169e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:15.458 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:15.458 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 160 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":472,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:15.472 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:15.472 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:15.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:15.579+0000 7f7676c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:15.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:15.586+0000 7f7676c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:15.773 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:15.773 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:15.785 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-04-16T07:53:15.161310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004136"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:15.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 262 16 1935 1919 0.45772 0 - 0.363217 2024-04-16T07:53:16.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 263 16 1935 1919 0.45598 0 - 0.363217 2024-04-16T07:53:17.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~218s 2024-04-16T07:53:17.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~191s 2024-04-16T07:53:17.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~218s 2024-04-16T07:53:17.777 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:17.777+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:17.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 264 16 1935 1919 0.454252 0 - 0.363217 2024-04-16T07:53:18.788 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:18.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 265 16 1935 1919 0.452538 0 - 0.363217 2024-04-16T07:53:18.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:18.908+0000 7f74832ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:18.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:18.916+0000 7f74832ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:19.099 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:19.099 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:19.111 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-04-16T07:53:17.162030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004237"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:19.113 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:19.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:19.219+0000 7f548760f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:19.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:19.226+0000 7f548760f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:19.477 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:19.477 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 160 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":476,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:19.490 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:19.490 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:19.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:19.599+0000 7ff90059d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:19.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:19.607+0000 7ff90059d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:19.789 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:19.789 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:19.802 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-04-16T07:53:19.162712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004185"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:19.902 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 266 16 1935 1919 0.450837 0 - 0.363217 2024-04-16T07:53:20.903 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 267 16 1935 1919 0.449148 0 - 0.363217 2024-04-16T07:53:21.903 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 268 16 1935 1919 0.447473 0 - 0.363217 2024-04-16T07:53:22.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~223s 2024-04-16T07:53:22.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~196s 2024-04-16T07:53:22.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~223s 2024-04-16T07:53:22.779 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:22.779+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:22.804 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:22.903 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 269 16 1935 1919 0.445809 0 - 0.363217 2024-04-16T07:53:22.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:22.924+0000 7f6e78ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:22.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:22.932+0000 7f6e78ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:23.116 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:23.128 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-04-16T07:53:21.163339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004072"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:23.130 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:23.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:23.237+0000 7f2fcd39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:23.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:23.244+0000 7f2fcd39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:23.495 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:23.495 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 165 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":480,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:23.508 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:23.509 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:23.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:23.616+0000 7f15cd864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:23.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:23.624+0000 7f15cd864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:23.806 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:23.807 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:23.819 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-04-16T07:53:23.164127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004139"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:23.903 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 270 16 1935 1919 0.444158 0 - 0.363217 2024-04-16T07:53:24.903 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 271 16 1935 1919 0.442519 0 - 0.363217 2024-04-16T07:53:25.903 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 272 16 1935 1919 0.440892 0 - 0.363217 2024-04-16T07:53:26.821 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:26.904 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 273 16 1935 1919 0.439277 0 - 0.363217 2024-04-16T07:53:26.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:26.942+0000 7f8b2bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:26.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:26.950+0000 7f8b2bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:27.133 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:27.134 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:27.146 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-16T07:53:25.164727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004076"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:27.148 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:27.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:27.258+0000 7f6259c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:27.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:27.264+0000 7f6259c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:27.519 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:27.519 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 170 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":484,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:27.533 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:27.533 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:27.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:27.641+0000 7f2e14354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:27.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:27.648+0000 7f2e14354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:27.780 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:27.780+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:27.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~229s 2024-04-16T07:53:27.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~202s 2024-04-16T07:53:27.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~229s 2024-04-16T07:53:27.832 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:27.832 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:27.845 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-04-16T07:53:27.165591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004281"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:27.904 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 274 16 1935 1919 0.437674 0 - 0.363217 2024-04-16T07:53:28.904 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 275 16 1935 1919 0.436082 0 - 0.363217 2024-04-16T07:53:29.904 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 276 16 1935 1919 0.434502 0 - 0.363217 2024-04-16T07:53:30.847 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:30.904 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 277 16 1935 1919 0.432934 0 - 0.363217 2024-04-16T07:53:30.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:30.968+0000 7f2bfda64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:30.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:30.976+0000 7f2bfda64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:31.160 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:31.161 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:31.173 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-04-16T07:53:29.166213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004183"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:31.174 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:31.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:31.282+0000 7ff94526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:31.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:31.288+0000 7ff94526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:31.539 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:31.539 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 175 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":488,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:31.553 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:31.553 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:31.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:31.661+0000 7f7a06d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:31.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:31.668+0000 7f7a06d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:31.851 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:31.852 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:31.864 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-04-16T07:53:31.166976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004264"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182803,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:31.904 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 278 16 1935 1919 0.431376 0 - 0.363217 2024-04-16T07:53:32.781 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:32.781+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:32.904 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 279 16 1935 1919 0.42983 0 - 0.363217 2024-04-16T07:53:33.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~234s 2024-04-16T07:53:33.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~207s 2024-04-16T07:53:33.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~234s 2024-04-16T07:53:33.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:53:33.905903+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:53:33.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:53:33.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 280 16 1935 1919 0.428295 0 - 0.363217 2024-04-16T07:53:34.866 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:34.904 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 281 16 1935 1919 0.426771 0 - 0.363217 2024-04-16T07:53:34.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:34.987+0000 7fd83986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:34.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:34.995+0000 7fd83986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:35.180 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:35.180 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:35.192 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-04-16T07:53:35.168453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004325"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182808,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:35.194 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:35.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:35.341+0000 7f72d2954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:35.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:35.347+0000 7f72d2954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:35.598 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:35.598 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 180 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":492,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:35.611 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:35.612 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:35.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:35.719+0000 7f007546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:35.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:35.727+0000 7f007546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:35.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 282 16 1935 1919 0.425258 0 - 0.363217 2024-04-16T07:53:35.910 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:35.910 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:35.923 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-04-16T07:53:35.168453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004325"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182808,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:36.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 283 16 1935 1919 0.423755 0 - 0.363217 2024-04-16T07:53:37.782 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:37.782+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:37.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 284 16 1935 1919 0.422263 0 - 0.363217 2024-04-16T07:53:38.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~239s 2024-04-16T07:53:38.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~212s 2024-04-16T07:53:38.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~239s 2024-04-16T07:53:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T07:53:38.832 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:53:38.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 285 16 1935 1919 0.420781 0 - 0.363217 2024-04-16T07:53:38.925 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:39.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:39.045+0000 7f5e7fe64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:39.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:39.053+0000 7f5e7fe64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:39.236 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:39.236 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:39.248 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-16T07:53:39.169789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004198"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182808,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:39.250 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:39.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:39.358+0000 7f34c6424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:39.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:39.364+0000 7f34c6424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:39.617 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:39.617 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 185 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":496,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:39.630 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:39.630 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:39.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:39.736+0000 7fe14acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:39.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:39.744+0000 7fe14acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:39.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 286 16 1935 1919 0.41931 0 - 0.363217 2024-04-16T07:53:39.929 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:39.930 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:39.942 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-16T07:53:39.169789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004198"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182808,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:40.833 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:53:40.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 287 16 1935 1919 0.417849 0 - 0.363217 2024-04-16T07:53:40.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:40.946+0000 7f6c7995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:40.952+0000 7f6c7995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-16T07:53:41.905 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 288 16 1935 1919 0.416398 0 - 0.363217 2024-04-16T07:53:42.783 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:42.783+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:42.906 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 289 16 1935 1919 0.414957 0 - 0.363217 2024-04-16T07:53:42.944 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:43.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:43.066+0000 7f9bd2064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:43.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:43.074+0000 7f9bd2064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:43.259 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:43.259 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:43.272 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-04-16T07:53:43.171345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004368"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:43.273 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:43.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:43.381+0000 7fa8b459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:43.387+0000 7fa8b459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:43.638 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:43.638 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 190 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":500,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:43.651 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:43.651 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:43.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~244s 2024-04-16T07:53:43.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~217s 2024-04-16T07:53:43.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~244s 2024-04-16T07:53:43.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:43.758+0000 7f91662ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:43.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:43.766+0000 7f91662ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:43.906 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 290 16 1935 1919 0.413526 0 - 0.363217 2024-04-16T07:53:43.951 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:43.951 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:43.964 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-04-16T07:53:43.171345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004368"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:44.906 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 291 16 1935 1919 0.412105 0 - 0.363217 2024-04-16T07:53:45.906 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 292 16 1935 1919 0.410694 0 - 0.363217 2024-04-16T07:53:46.906 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 293 16 1935 1919 0.409292 0 - 0.363217 2024-04-16T07:53:46.966 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:47.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:47.086+0000 7ff53d02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:47.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:47.093+0000 7ff53d02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:47.278 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:47.279 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:47.291 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-04-16T07:53:47.172860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004407"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:47.293 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:47.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:47.400+0000 7f8126e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:47.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:47.407+0000 7f8126e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:47.660 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:47.660 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 190 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":504,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:47.673 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:47.673 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:47.783 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:47.783+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:47.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:47.782+0000 7f475feed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:47.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:47.789+0000 7f475feed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:47.906 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 294 16 1935 1919 0.4079 0 - 0.363217 2024-04-16T07:53:47.973 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:47.974 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:47.986 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-04-16T07:53:47.172860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004407"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:48.906 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 295 16 1935 1919 0.406517 0 - 0.363217 2024-04-16T07:53:49.037 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~250s 2024-04-16T07:53:49.037 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~223s 2024-04-16T07:53:49.037 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~250s 2024-04-16T07:53:49.906 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 296 16 1935 1919 0.405144 0 - 0.363217 2024-04-16T07:53:50.906 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 297 16 1935 1919 0.40378 0 - 0.363217 2024-04-16T07:53:50.988 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:51.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:51.110+0000 7fc1e842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:51.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:51.118+0000 7fc1e842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:51.302 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:51.303 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:51.315 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-04-16T07:53:51.174269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004479"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182823,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:51.316 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:51.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:51.426+0000 7feb3006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:51.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:51.432+0000 7feb3006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:51.683 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:51.684 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 195 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":508,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:51.696 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:51.696 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:51.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:51.804+0000 7f2ffc217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:51.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:51.812+0000 7f2ffc217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:51.907 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 298 16 1935 1919 0.402425 0 - 0.363217 2024-04-16T07:53:51.995 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:51.996 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:52.009 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-04-16T07:53:51.174269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004479"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182823,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:52.784 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:52.784+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:52.907 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 299 16 1935 1919 0.401079 0 - 0.363217 2024-04-16T07:53:53.907 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:53:53.908440+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:53:53.907 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:53:53.907 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 300 16 1935 1919 0.399742 0 - 0.363217 2024-04-16T07:53:54.339 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2024-04-16T07:53:54.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~228s 2024-04-16T07:53:54.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~255s 2024-04-16T07:53:54.907 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 301 16 1935 1919 0.398414 0 - 0.363217 2024-04-16T07:53:55.010 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:55.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:55.130+0000 7f18d566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:55.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:55.138+0000 7f18d566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:55.324 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:55.324 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:55.337 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-04-16T07:53:55.175631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004285"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182828,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:55.338 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:55.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:55.445+0000 7f1e2399d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:55.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:55.452+0000 7f1e2399d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:55.707 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:55.707 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 200 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":512,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:55.720 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:55.720 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:55.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:55.829+0000 7f285dd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:55.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:55.836+0000 7f285dd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:55.907 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 302 16 1935 1919 0.397095 0 - 0.363217 2024-04-16T07:53:56.019 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:56.020 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:56.032 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-04-16T07:53:55.175631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004285"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182828,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:56.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 303 16 1935 1919 0.395784 0 - 0.363217 2024-04-16T07:53:57.784 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:53:57.785+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:53:57.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 304 16 1935 1919 0.394482 0 - 0.363217 2024-04-16T07:53:58.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 305 16 1935 1919 0.393189 0 - 0.363217 2024-04-16T07:53:59.035 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:59.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:59.156+0000 7fad61a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:59.164+0000 7fad61a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:59.347 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:59.348 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:53:59.360 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-16T07:53:59.177003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004143"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182828,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:53:59.361 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:53:59.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:59.471+0000 7f3e96464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:59.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:59.477+0000 7f3e96464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:59.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2024-04-16T07:53:59.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~233s 2024-04-16T07:53:59.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~260s 2024-04-16T07:53:59.727 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:53:59.727 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 200 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":517,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:53:59.740 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:53:59.740 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:53:59.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:59.850+0000 7fa0bc464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:59.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:53:59.857+0000 7fa0bc464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:53:59.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 306 16 1935 1919 0.391904 0 - 0.363217 2024-04-16T07:54:00.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:00.042 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:00.055 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-16T07:53:59.177003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004143"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182828,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:00.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 307 16 1935 1919 0.390627 0 - 0.363217 2024-04-16T07:54:01.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 308 16 1935 1919 0.389359 0 - 0.363217 2024-04-16T07:54:02.785 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:02.785+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:54:02.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 309 16 1935 1919 0.388099 0 - 0.363217 2024-04-16T07:54:03.057 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:03.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:03.178+0000 7f1aaa6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:03.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:03.186+0000 7f1aaa6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:03.370 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:03.371 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:03.383 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-04-16T07:54:03.178503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004233"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:03.385 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:03.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:03.493+0000 7f76b226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:03.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:03.499+0000 7f76b226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:03.749 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:03.749 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 205 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":521,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:03.762 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:03.763 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:03.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:03.871+0000 7f63a1e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:03.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:03.879+0000 7f63a1e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:03.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 310 16 1935 1919 0.386847 0 - 0.363217 2024-04-16T07:54:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:04.063 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:04.075 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-04-16T07:54:03.178503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004233"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:04.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 311 16 1935 1919 0.385603 0 - 0.363217 2024-04-16T07:54:04.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~266s 2024-04-16T07:54:04.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~239s 2024-04-16T07:54:04.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~266s 2024-04-16T07:54:05.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 312 16 1935 1919 0.384367 0 - 0.363217 2024-04-16T07:54:06.909 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 313 16 1935 1919 0.383139 0 - 0.363217 2024-04-16T07:54:07.078 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:07.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:07.198+0000 7f954c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:07.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:07.206+0000 7f954c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:07.389 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:07.390 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:07.402 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-04-16T07:54:07.179999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004367"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:07.403 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:07.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:07.510+0000 7f988266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:07.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:07.517+0000 7f988266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:07.772 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:07.773 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 210 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":525,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:07.786 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:07.786 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:07.787 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:07.786+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:54:07.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:07.894+0000 7f85b466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:07.902+0000 7f85b466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:07.908 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 314 16 1935 1919 0.381919 0 - 0.363217 2024-04-16T07:54:08.084 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:08.085 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:08.097 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-04-16T07:54:07.179999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004367"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:08.909 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 315 16 1935 1919 0.380707 0 - 0.363217 2024-04-16T07:54:09.909 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 316 16 1935 1919 0.379502 0 - 0.363217 2024-04-16T07:54:10.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~271s 2024-04-16T07:54:10.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~244s 2024-04-16T07:54:10.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~271s 2024-04-16T07:54:10.909 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 317 16 1935 1919 0.378305 0 - 0.363217 2024-04-16T07:54:11.099 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:11.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:11.220+0000 7feef3e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:11.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:11.227+0000 7feef3e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:11.413 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:11.426 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-04-16T07:54:11.181461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004457"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:11.427 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:11.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:11.535+0000 7f2dfb66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:11.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:11.541+0000 7f2dfb66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:11.794 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:11.795 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 215 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":529,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:11.808 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:11.808 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:11.909 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 318 16 1935 1919 0.377115 0 - 0.363217 2024-04-16T07:54:11.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:11.916+0000 7fe26e613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:11.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:11.924+0000 7fe26e613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:12.107 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:12.107 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:12.119 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-04-16T07:54:11.181461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004457"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:12.787 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:12.788+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:54:12.909 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 319 16 1935 1919 0.375933 0 - 0.363217 2024-04-16T07:54:13.909 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:54:13.910845+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:54:13.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:54:13.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 320 16 1935 1919 0.374758 0 - 0.363217 2024-04-16T07:54:14.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 321 16 1935 1919 0.373591 0 - 0.363217 2024-04-16T07:54:15.121 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:15.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:15.242+0000 7fd44506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:15.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:15.250+0000 7fd44506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:15.433 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:15.433 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:15.445 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-04-16T07:54:15.182816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004312"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:15.447 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:15.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~276s 2024-04-16T07:54:15.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~249s 2024-04-16T07:54:15.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~276s 2024-04-16T07:54:15.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:15.555+0000 7f756dfc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:15.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:15.562+0000 7f756dfc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:15.812 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:15.812 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 220 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":533,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:15.825 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:15.825 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:15.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 322 16 1935 1919 0.372431 0 - 0.363217 2024-04-16T07:54:15.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:15.932+0000 7fdd35f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:15.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:15.939+0000 7fdd35f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:16.124 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:16.136 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-04-16T07:54:15.182816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004312"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:16.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 323 16 1935 1919 0.371278 0 - 0.363217 2024-04-16T07:54:17.787 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:17.787+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:54:17.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 324 16 1935 1919 0.370132 0 - 0.363217 2024-04-16T07:54:18.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 325 16 1935 1919 0.368993 0 - 0.363217 2024-04-16T07:54:19.139 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:19.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:19.260+0000 7f5c1d59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:19.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:19.268+0000 7f5c1d59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:19.451 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:19.452 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:19.464 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-04-16T07:54:19.184146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004147"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:19.466 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:19.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:19.573+0000 7fa708aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:19.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:19.579+0000 7fa708aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:19.831 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:19.831 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 220 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":537,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:19.844 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:19.845 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:19.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 326 16 1935 1919 0.367861 0 - 0.363217 2024-04-16T07:54:19.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:19.953+0000 7f70669ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:19.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:19.960+0000 7f706579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:20.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:20.145 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:20.157 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-04-16T07:54:19.184146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004147"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:20.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~282s 2024-04-16T07:54:20.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~255s 2024-04-16T07:54:20.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~282s 2024-04-16T07:54:20.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 327 16 1935 1919 0.366736 0 - 0.363217 2024-04-16T07:54:21.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 328 16 1935 1919 0.365618 0 - 0.363217 2024-04-16T07:54:22.788 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:22.788+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:54:22.910 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 329 16 1935 1919 0.364506 0 - 0.363217 2024-04-16T07:54:23.159 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:23.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:23.280+0000 7f2b29c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:23.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:23.288+0000 7f2b29c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:23.471 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:23.472 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:23.484 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-16T07:54:23.185397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003935"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:23.487 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:23.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:23.593+0000 7f7d75c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:23.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:23.600+0000 7f7d75c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:23.849 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:23.850 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 230 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":541,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:23.863 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:23.863 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:23.911 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 330 16 1935 1919 0.363402 0 - 0.363217 2024-04-16T07:54:23.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:23.970+0000 7fdba9b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:23.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:23.978+0000 7fdba9b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:24.162 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:24.162 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:24.175 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-16T07:54:23.185397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003935"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:24.911 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 331 16 1935 1919 0.362304 0 - 0.363217 2024-04-16T07:54:25.911 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 332 16 1935 1919 0.361213 0 - 0.363217 2024-04-16T07:54:26.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~287s 2024-04-16T07:54:26.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~260s 2024-04-16T07:54:26.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~287s 2024-04-16T07:54:26.911 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 333 16 1935 1919 0.360128 0 - 0.363217 2024-04-16T07:54:27.177 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:27.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:27.298+0000 7f89dd15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:27.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:27.306+0000 7f89dd15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:27.492 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:27.493 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:27.505 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-04-16T07:54:27.186855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004039"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:27.507 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:27.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:27.615+0000 7fd59b39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:27.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:27.622+0000 7fd59b39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:27.788 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:27.788+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:54:27.875 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:27.875 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 230 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":545,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:27.888 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:27.888 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:27.911 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 334 16 1935 1919 0.35905 0 - 0.363217 2024-04-16T07:54:27.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:27.995+0000 7f886306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:28.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:28.003+0000 7f886306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:28.188 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:28.188 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:28.201 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-04-16T07:54:27.186855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004039"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:28.911 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 335 16 1935 1919 0.357978 0 - 0.363217 2024-04-16T07:54:29.912 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 336 16 1935 1919 0.356913 0 - 0.363217 2024-04-16T07:54:30.912 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 337 16 1935 1919 0.355853 0 - 0.363217 2024-04-16T07:54:31.203 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:31.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:31.325+0000 7fcbf499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:31.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:31.332+0000 7fcbf499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:31.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~292s 2024-04-16T07:54:31.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2024-04-16T07:54:31.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~292s 2024-04-16T07:54:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:31.517 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:31.529 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-16T07:54:31.188295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004149"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:31.531 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:31.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:31.639+0000 7f1e1f8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:31.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:31.645+0000 7f1e1f8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:31.896 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 235 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":549,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:31.909 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:31.909 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:31.914 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 338 16 1935 1919 0.354801 0 - 0.363217 2024-04-16T07:54:32.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:32.017+0000 7f48760ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:32.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:32.024+0000 7f48760ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:32.210 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:32.210 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:32.222 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-16T07:54:31.188295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004149"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:32.789 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:32.789+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:54:32.912 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 339 16 1935 1919 0.353754 0 - 0.363217 2024-04-16T07:54:33.912 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:54:33.913425+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:54:33.912 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:54:33.912 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 340 16 1935 1919 0.352714 0 - 0.363217 2024-04-16T07:54:34.912 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 341 16 1935 1919 0.351679 0 - 0.363217 2024-04-16T07:54:35.224 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:35.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:35.345+0000 7f706086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:35.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:35.353+0000 7f706086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:35.537 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:35.537 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:35.549 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-04-16T07:54:35.189683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004286"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:35.551 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:35.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:35.658+0000 7fceecead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:35.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:35.664+0000 7fceecead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:35.912 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 342 16 1935 1919 0.350651 0 - 0.363217 2024-04-16T07:54:35.915 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:35.915 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 240 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":553,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:35.928 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:35.929 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:36.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:36.036+0000 7f131a35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:36.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:36.044+0000 7f131a35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:36.225 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:36.226 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:36.238 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-04-16T07:54:35.189683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004286"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:36.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~297s 2024-04-16T07:54:36.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2024-04-16T07:54:36.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~297s 2024-04-16T07:54:36.912 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 343 16 1935 1919 0.349629 0 - 0.363217 2024-04-16T07:54:37.790 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:37.790+0000 7fa0cfff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.79:6806/4105421492 for 29sec e61 v61) 2024-04-16T07:54:37.913 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 344 16 1935 1919 0.348612 0 - 0.363217 2024-04-16T07:54:38.913 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 345 16 1935 1919 0.347602 0 - 0.363217 2024-04-16T07:54:39.240 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:39.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:39.361+0000 7f93f5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:39.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:39.369+0000 7f93f5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:39.550 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:39.551 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:39.563 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-16T07:54:39.191212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004357"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:39.564 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-16T07:54:39.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:39.672+0000 7fc99286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:39.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:39.678+0000 7fc99286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:39.913 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 346 16 1935 1919 0.346597 0 - 0.363217 2024-04-16T07:54:39.931 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:39.931 INFO:teuthology.orchestra.run.smithi079.stdout:{"fsid":"8dbe1cb9-e93e-4590-9181-7b27048c6254","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 12 pgs inactive, 5 pgs peering, 5 pgs stale","count":22},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 605/2949 objects degraded (20.515%), 8 pgs degraded, 7 pgs undersized","count":15},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 245 sec, mon.a has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":557,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":2,"osd_up_since":1713253804,"num_in_osds":4,"osd_in_since":1713253524,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":5},{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+degraded","count":4},{"state_name":"stale+active+clean","count":3},{"state_name":"peering","count":3},{"state_name":"undersized+degraded+peered","count":2},{"state_name":"active+recovering+degraded","count":2},{"state_name":"stale+peering","count":2}],"num_pgs":25,"num_pools":3,"num_objects":983,"data_bytes":64421888,"bytes_used":149184512,"bytes_avail":386397872128,"bytes_total":386547056640,"inactive_pgs_ratio":0.47999998927116394,"degraded_objects":605,"degraded_total":2949,"degraded_ratio":0.20515428958969142},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-16T07:46:29.047986+0000","services":{}},"progress_events":{}} 2024-04-16T07:54:39.945 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-16T07:54:39.945 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:40.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:40.053+0000 7fa96602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:40.060+0000 7fa96602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:40.245 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-16T07:54:40.245 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-16T07:54:40.257 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-16T07:54:39.191212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":145688,"kb_used_data":21944,"kb_used_omap":0,"kb_used_meta":123712,"kb_avail":377341672,"statfs":{"total":386547056640,"available":386397872128,"internally_reserved":0,"allocated":22470656,"data_stored":329444371,"data_compressed":15612336,"data_compressed_allocated":20557824,"data_compressed_original":328679424,"omap_allocated":0,"internal_metadata":126681088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004357"},"pg_stats":[{"pgid":"4.f","version":"55'116","reported_seq":168,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.262284+0000","last_change":"2024-04-16T07:50:05.261429+0000","last_active":"2024-04-16T07:50:04.269432+0000","last_peered":"2024-04-16T07:50:05.262284+0000","last_clean":"2024-04-16T07:49:53.209226+0000","last_became_active":"2024-04-16T07:49:22.880458+0000","last_became_peered":"2024-04-16T07:50:05.261429+0000","last_unstale":"2024-04-16T07:50:05.262284+0000","last_undegraded":"2024-04-16T07:50:05.240813+0000","last_fullsized":"2024-04-16T07:50:05.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"55'117","reported_seq":207,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.217658+0000","last_change":"2024-04-16T07:50:04.217658+0000","last_active":"2024-04-16T07:50:04.217658+0000","last_peered":"2024-04-16T07:49:53.200689+0000","last_clean":"2024-04-16T07:49:53.200689+0000","last_became_active":"2024-04-16T07:49:35.899630+0000","last_became_peered":"2024-04-16T07:49:35.899630+0000","last_unstale":"2024-04-16T07:50:04.217658+0000","last_undegraded":"2024-04-16T07:50:04.217658+0000","last_fullsized":"2024-04-16T07:50:04.217658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"55'102","reported_seq":119,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.921998+0000","last_change":"2024-04-16T07:49:36.137708+0000","last_active":"2024-04-16T07:49:55.921998+0000","last_peered":"2024-04-16T07:49:55.921998+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.936444+0000","last_became_peered":"2024-04-16T07:49:35.936444+0000","last_unstale":"2024-04-16T07:49:55.921998+0000","last_undegraded":"2024-04-16T07:49:35.787961+0000","last_fullsized":"2024-04-16T07:49:55.921998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,1,2","objects":3},{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"48'105","reported_seq":119,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.822703+0000","last_change":"2024-04-16T07:49:21.822703+0000","last_active":"2024-04-16T07:49:21.822703+0000","last_peered":"2024-04-16T07:49:02.184088+0000","last_clean":"2024-04-16T07:49:02.184088+0000","last_became_active":"2024-04-16T07:48:52.945051+0000","last_became_peered":"2024-04-16T07:48:52.945051+0000","last_unstale":"2024-04-16T07:50:04.193564+0000","last_undegraded":"2024-04-16T07:49:21.822703+0000","last_fullsized":"2024-04-16T07:49:21.822703+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"48'101","reported_seq":122,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.788948+0000","last_change":"2024-04-16T07:48:52.982210+0000","last_active":"2024-04-16T07:49:22.788948+0000","last_peered":"2024-04-16T07:49:22.788948+0000","last_clean":"2024-04-16T07:49:22.788948+0000","last_became_active":"2024-04-16T07:48:52.976146+0000","last_became_peered":"2024-04-16T07:48:52.976146+0000","last_unstale":"2024-04-16T07:50:04.193570+0000","last_undegraded":"2024-04-16T07:49:22.788948+0000","last_fullsized":"2024-04-16T07:49:22.788948+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"53'107","reported_seq":162,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.942058+0000","last_change":"2024-04-16T07:49:36.496690+0000","last_active":"2024-04-16T07:49:55.942058+0000","last_peered":"2024-04-16T07:49:55.942058+0000","last_clean":"2024-04-16T07:49:55.942058+0000","last_became_active":"2024-04-16T07:49:35.919244+0000","last_became_peered":"2024-04-16T07:49:35.919244+0000","last_unstale":"2024-04-16T07:49:55.942058+0000","last_undegraded":"2024-04-16T07:49:55.942058+0000","last_fullsized":"2024-04-16T07:49:55.942058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"55'127","reported_seq":136,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.939341+0000","last_change":"2024-04-16T07:49:36.020139+0000","last_active":"2024-04-16T07:49:55.939341+0000","last_peered":"2024-04-16T07:49:55.939341+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:36.007569+0000","last_became_peered":"2024-04-16T07:49:36.007569+0000","last_unstale":"2024-04-16T07:49:55.939341+0000","last_undegraded":"2024-04-16T07:49:35.822546+0000","last_fullsized":"2024-04-16T07:49:55.939341+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":-20},{"shards":"0,3","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"55'111","reported_seq":117,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.940972+0000","last_change":"2024-04-16T07:49:35.979748+0000","last_active":"2024-04-16T07:49:55.940972+0000","last_peered":"2024-04-16T07:49:55.940972+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.967610+0000","last_became_peered":"2024-04-16T07:49:35.967610+0000","last_unstale":"2024-04-16T07:49:55.940972+0000","last_undegraded":"2024-04-16T07:49:35.809885+0000","last_fullsized":"2024-04-16T07:49:55.940972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":26,"num_objects_missing":26,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,1,3","objects":1},{"shards":"0,3","objects":26}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'122","reported_seq":134,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934034+0000","last_change":"2024-04-16T07:49:35.984828+0000","last_active":"2024-04-16T07:49:55.934034+0000","last_peered":"2024-04-16T07:49:55.934034+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.979926+0000","last_became_peered":"2024-04-16T07:49:35.979926+0000","last_unstale":"2024-04-16T07:49:55.934034+0000","last_undegraded":"2024-04-16T07:49:35.798853+0000","last_fullsized":"2024-04-16T07:49:55.934034+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,3","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.268722+0000","last_change":"2024-04-16T07:50:05.267925+0000","last_active":"2024-04-16T07:45:32.048077+0000","last_peered":"2024-04-16T07:50:05.268722+0000","last_clean":"2024-04-16T07:45:32.048077+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.267925+0000","last_unstale":"2024-04-16T07:50:05.268722+0000","last_undegraded":"2024-04-16T07:50:05.268722+0000","last_fullsized":"2024-04-16T07:50:05.236486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:32.048077+0000","last_clean_scrub_stamp":"2024-04-16T07:45:32.048077+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.926320+0000","last_change":"2024-04-16T07:49:35.972990+0000","last_active":"2024-04-16T07:49:55.926320+0000","last_peered":"2024-04-16T07:49:55.926320+0000","last_clean":"2024-04-16T07:49:55.926320+0000","last_became_active":"2024-04-16T07:49:35.966074+0000","last_became_peered":"2024-04-16T07:49:35.966074+0000","last_unstale":"2024-04-16T07:49:55.926320+0000","last_undegraded":"2024-04-16T07:49:55.926320+0000","last_fullsized":"2024-04-16T07:49:55.926320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"54'135","reported_seq":8,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.271543+0000","last_change":"2024-04-16T07:50:05.270780+0000","last_active":"2024-04-16T07:48:51.768206+0000","last_peered":"2024-04-16T07:50:05.271543+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"0.000000","last_became_peered":"2024-04-16T07:50:05.270780+0000","last_unstale":"2024-04-16T07:50:05.271543+0000","last_undegraded":"2024-04-16T07:50:05.271543+0000","last_fullsized":"2024-04-16T07:50:05.248538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.790566+0000","last_change":"2024-04-16T07:48:47.924548+0000","last_active":"2024-04-16T07:49:22.790566+0000","last_peered":"2024-04-16T07:49:22.790566+0000","last_clean":"2024-04-16T07:49:22.790566+0000","last_became_active":"2024-04-16T07:48:47.917875+0000","last_became_peered":"2024-04-16T07:48:47.917875+0000","last_unstale":"2024-04-16T07:50:04.193596+0000","last_undegraded":"2024-04-16T07:49:22.790566+0000","last_fullsized":"2024-04-16T07:49:22.790566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"53'120","reported_seq":172,"reported_epoch":61,"state":"undersized+degraded+peered","last_fresh":"2024-04-16T07:50:05.245733+0000","last_change":"2024-04-16T07:50:05.244801+0000","last_active":"2024-04-16T07:50:04.231462+0000","last_peered":"2024-04-16T07:50:05.245733+0000","last_clean":"2024-04-16T07:49:53.190048+0000","last_became_active":"2024-04-16T07:49:22.852725+0000","last_became_peered":"2024-04-16T07:50:05.244801+0000","last_unstale":"2024-04-16T07:50:05.245733+0000","last_undegraded":"2024-04-16T07:50:05.229108+0000","last_fullsized":"2024-04-16T07:50:05.227802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":120}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.916323+0000","last_change":"2024-04-16T07:49:35.898141+0000","last_active":"2024-04-16T07:49:55.916323+0000","last_peered":"2024-04-16T07:49:55.916323+0000","last_clean":"2024-04-16T07:49:55.916323+0000","last_became_active":"2024-04-16T07:49:35.887671+0000","last_became_peered":"2024-04-16T07:49:35.887671+0000","last_unstale":"2024-04-16T07:49:55.916323+0000","last_undegraded":"2024-04-16T07:49:55.916323+0000","last_fullsized":"2024-04-16T07:49:55.916323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"55'120","reported_seq":170,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.219649+0000","last_change":"2024-04-16T07:50:04.219649+0000","last_active":"2024-04-16T07:50:04.219649+0000","last_peered":"2024-04-16T07:49:53.197048+0000","last_clean":"2024-04-16T07:49:02.360526+0000","last_became_active":"2024-04-16T07:49:35.933666+0000","last_became_peered":"2024-04-16T07:49:35.933666+0000","last_unstale":"2024-04-16T07:50:04.219649+0000","last_undegraded":"2024-04-16T07:50:04.219649+0000","last_fullsized":"2024-04-16T07:50:04.219649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":120,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-16T07:50:04.263684+0000","last_change":"2024-04-16T07:49:34.800285+0000","last_active":"2024-04-16T07:49:34.800285+0000","last_peered":"2024-04-16T07:49:33.308545+0000","last_clean":"2024-04-16T07:49:22.801972+0000","last_became_active":"2024-04-16T07:49:33.302478+0000","last_became_peered":"2024-04-16T07:49:33.302478+0000","last_unstale":"2024-04-16T07:50:04.263684+0000","last_undegraded":"2024-04-16T07:50:04.263684+0000","last_fullsized":"2024-04-16T07:50:04.263684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"55'133","reported_seq":144,"reported_epoch":59,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-16T07:49:55.934780+0000","last_change":"2024-04-16T07:49:36.137876+0000","last_active":"2024-04-16T07:49:55.934780+0000","last_peered":"2024-04-16T07:49:55.934780+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.897850+0000","last_became_peered":"2024-04-16T07:49:35.897850+0000","last_unstale":"2024-04-16T07:49:55.934780+0000","last_undegraded":"2024-04-16T07:49:35.803344+0000","last_fullsized":"2024-04-16T07:49:55.934780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":28,"num_objects_missing":28,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":-27},{"shards":"2,3","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.249235+0000","last_change":"2024-04-16T07:50:05.248415+0000","last_active":"2024-04-16T07:50:04.241461+0000","last_peered":"2024-04-16T07:50:05.249235+0000","last_clean":"2024-04-16T07:49:53.194258+0000","last_became_active":"2024-04-16T07:49:22.847154+0000","last_became_peered":"2024-04-16T07:50:05.248415+0000","last_unstale":"2024-04-16T07:50:05.249235+0000","last_undegraded":"2024-04-16T07:50:05.249235+0000","last_fullsized":"2024-04-16T07:50:05.224017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"48'105","reported_seq":126,"reported_epoch":51,"state":"stale+active+clean","last_fresh":"2024-04-16T07:49:22.787540+0000","last_change":"2024-04-16T07:48:52.977680+0000","last_active":"2024-04-16T07:49:22.787540+0000","last_peered":"2024-04-16T07:49:22.787540+0000","last_clean":"2024-04-16T07:49:22.787540+0000","last_became_active":"2024-04-16T07:48:52.965123+0000","last_became_peered":"2024-04-16T07:48:52.965123+0000","last_unstale":"2024-04-16T07:50:04.193601+0000","last_undegraded":"2024-04-16T07:49:22.787540+0000","last_fullsized":"2024-04-16T07:49:22.787540+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.237025+0000","last_change":"2024-04-16T07:50:05.236154+0000","last_active":"2024-04-16T07:49:21.782628+0000","last_peered":"2024-04-16T07:50:05.237025+0000","last_clean":"2024-04-16T07:48:22.899313+0000","last_became_active":"2024-04-16T07:48:22.892718+0000","last_became_peered":"2024-04-16T07:50:05.236154+0000","last_unstale":"2024-04-16T07:50:05.237025+0000","last_undegraded":"2024-04-16T07:50:05.237025+0000","last_fullsized":"2024-04-16T07:50:05.220931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"48'109","reported_seq":124,"reported_epoch":51,"state":"stale+peering","last_fresh":"2024-04-16T07:49:21.774567+0000","last_change":"2024-04-16T07:49:20.800674+0000","last_active":"2024-04-16T07:49:20.800674+0000","last_peered":"2024-04-16T07:49:02.267479+0000","last_clean":"2024-04-16T07:49:02.267479+0000","last_became_active":"2024-04-16T07:48:52.957781+0000","last_became_peered":"2024-04-16T07:48:52.957781+0000","last_unstale":"2024-04-16T07:50:04.193606+0000","last_undegraded":"2024-04-16T07:49:21.774567+0000","last_fullsized":"2024-04-16T07:49:21.774567+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":61,"state":"undersized+peered","last_fresh":"2024-04-16T07:50:05.264907+0000","last_change":"2024-04-16T07:50:05.264162+0000","last_active":"2024-04-16T07:50:04.256242+0000","last_peered":"2024-04-16T07:50:05.264907+0000","last_clean":"2024-04-16T07:49:53.211073+0000","last_became_active":"2024-04-16T07:49:22.885743+0000","last_became_peered":"2024-04-16T07:50:05.264162+0000","last_unstale":"2024-04-16T07:50:05.264907+0000","last_undegraded":"2024-04-16T07:50:05.264907+0000","last_fullsized":"2024-04-16T07:50:05.243917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"53'114","reported_seq":132,"reported_epoch":59,"state":"active+recovering+degraded","last_fresh":"2024-04-16T07:49:55.936687+0000","last_change":"2024-04-16T07:49:36.226000+0000","last_active":"2024-04-16T07:49:55.936687+0000","last_peered":"2024-04-16T07:49:55.936687+0000","last_clean":"2024-04-16T07:48:51.768206+0000","last_became_active":"2024-04-16T07:49:35.988878+0000","last_became_peered":"2024-04-16T07:49:35.988878+0000","last_unstale":"2024-04-16T07:49:55.936687+0000","last_undegraded":"2024-04-16T07:49:35.814191+0000","last_fullsized":"2024-04-16T07:49:55.936687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:48:51.768206+0000","last_clean_scrub_stamp":"2024-04-16T07:48:51.768206+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":16,"num_objects_missing":16,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":16},{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-16T07:49:55.938208+0000","last_change":"2024-04-16T07:49:35.962339+0000","last_active":"2024-04-16T07:49:55.938208+0000","last_peered":"2024-04-16T07:49:55.938208+0000","last_clean":"2024-04-16T07:49:55.938208+0000","last_became_active":"2024-04-16T07:49:35.952144+0000","last_became_peered":"2024-04-16T07:49:35.952144+0000","last_unstale":"2024-04-16T07:49:55.938208+0000","last_undegraded":"2024-04-16T07:49:55.938208+0000","last_fullsized":"2024-04-16T07:49:55.938208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:45:33.051603+0000","last_clean_scrub_stamp":"2024-04-16T07:45:33.051603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":131,"num_objects_missing":131,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1844,"ondisk_log_size":1844,"up":40,"acting":40,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":17,"acting":17,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":50,"seq":214748364809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41956,"kb_used_data":5596,"kb_used_omap":0,"kb_used_meta":36352,"kb_avail":94329884,"statfs":{"total":96636764160,"available":96593801216,"internally_reserved":0,"allocated":5730304,"data_stored":84639738,"data_compressed":4012426,"data_compressed_allocated":5283840,"data_compressed_original":84459520,"omap_allocated":0,"internal_metadata":37224448},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":25,"seq":107374182868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32672,"kb_used_data":5336,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":5464064,"data_stored":79103195,"data_compressed":3748089,"data_compressed_allocated":4935680,"data_compressed_original":78888960,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":54,"seq":231928234008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38272,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94333568,"statfs":{"total":96636764160,"available":96597573632,"internally_reserved":0,"allocated":5955584,"data_stored":87615300,"data_compressed":4150506,"data_compressed_allocated":5464064,"data_compressed_original":87425024,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32788,"kb_used_data":5196,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":5320704,"data_stored":78086138,"data_compressed":3701315,"data_compressed_allocated":4874240,"data_compressed_original":77905920,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:54:40.913 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 347 16 1935 1919 0.345598 0 - 0.363217 2024-04-16T07:54:41.913 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 348 16 1935 1919 0.344605 0 - 0.363217 2024-04-16T07:54:42.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~303s 2024-04-16T07:54:42.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~276s 2024-04-16T07:54:42.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~303s 2024-04-16T07:54:42.072 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-16T07:54:42.376 INFO:tasks.ceph.osd.2:Sent signal 15 2024-04-16T07:54:42.377 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-16T07:54:42.377 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-16T07:54:42.378 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:42.378+0000 7fa0d2fff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 29729) UID: 0 2024-04-16T07:54:42.378 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-16T07:54:42.378+0000 7fa0d2fff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T07:54:42.378 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:54:42.377+0000 7fc5b6df7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 29154) UID: 0 2024-04-16T07:54:42.378 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-16T07:54:42.377+0000 7fc5b6df7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-16T07:54:42.913 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 349 16 1935 1919 0.343618 0 - 0.363217 2024-04-16T07:54:43.260 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:54:43.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:54:43.381+0000 7fb03ef1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:54:43.913 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 350 16 1935 1919 0.342636 0 - 0.363217 2024-04-16T07:54:44.913 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 351 16 1935 1919 0.34166 0 - 0.363217 2024-04-16T07:54:45.913 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 352 16 1935 1919 0.340689 0 - 0.363217 2024-04-16T07:54:46.914 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 353 16 1935 1919 0.339724 0 - 0.363217 2024-04-16T07:54:47.914 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 354 16 1935 1919 0.338764 0 - 0.363217 2024-04-16T07:54:48.914 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 355 16 1935 1919 0.33781 0 - 0.363217 2024-04-16T07:54:49.914 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 356 16 1935 1919 0.336861 0 - 0.363217 2024-04-16T07:54:50.914 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 357 16 1935 1919 0.335918 0 - 0.363217 2024-04-16T07:54:51.914 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 358 16 1935 1919 0.334979 0 - 0.363217 2024-04-16T07:54:52.914 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 359 16 1935 1919 0.334046 0 - 0.363217 2024-04-16T07:54:53.914 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:54:53.915917+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:54:53.915 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:54:53.915 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 360 16 1935 1919 0.333118 0 - 0.363217 2024-04-16T07:54:54.915 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 361 16 1935 1919 0.332196 0 - 0.363217 2024-04-16T07:54:55.915 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 362 16 1935 1919 0.331278 0 - 0.363217 2024-04-16T07:54:56.915 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 363 16 1935 1919 0.330365 0 - 0.363217 2024-04-16T07:54:57.915 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 364 16 1935 1919 0.329458 0 - 0.363217 2024-04-16T07:54:58.915 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 365 16 1935 1919 0.328555 0 - 0.363217 2024-04-16T07:54:59.915 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 366 16 1935 1919 0.327657 0 - 0.363217 2024-04-16T07:55:00.915 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 367 16 1935 1919 0.326765 0 - 0.363217 2024-04-16T07:55:01.916 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 368 16 1935 1919 0.325877 0 - 0.363217 2024-04-16T07:55:02.916 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 369 16 1935 1919 0.324993 0 - 0.363217 2024-04-16T07:55:03.916 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 370 16 1935 1919 0.324115 0 - 0.363217 2024-04-16T07:55:04.916 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 371 16 1935 1919 0.323241 0 - 0.363217 2024-04-16T07:55:05.916 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 372 16 1935 1919 0.322372 0 - 0.363217 2024-04-16T07:55:06.916 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 373 16 1935 1919 0.321508 0 - 0.363217 2024-04-16T07:55:07.916 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 374 16 1935 1919 0.320649 0 - 0.363217 2024-04-16T07:55:08.917 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 375 16 1935 1919 0.319793 0 - 0.363217 2024-04-16T07:55:09.917 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 376 16 1935 1919 0.318943 0 - 0.363217 2024-04-16T07:55:10.917 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 377 16 1935 1919 0.318097 0 - 0.363217 2024-04-16T07:55:11.917 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 378 16 1935 1919 0.317255 0 - 0.363217 2024-04-16T07:55:12.917 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 379 16 1935 1919 0.316418 0 - 0.363217 2024-04-16T07:55:13.917 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:55:13.918678+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:55:13.918 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:55:13.918 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 380 16 1935 1919 0.315586 0 - 0.363217 2024-04-16T07:55:14.917 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 381 16 1935 1919 0.314757 0 - 0.363217 2024-04-16T07:55:15.918 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 382 16 1935 1919 0.313933 0 - 0.363217 2024-04-16T07:55:16.918 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 383 16 1935 1919 0.313114 0 - 0.363217 2024-04-16T07:55:17.918 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 384 16 1935 1919 0.312298 0 - 0.363217 2024-04-16T07:55:18.918 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 385 16 1935 1919 0.311487 0 - 0.363217 2024-04-16T07:55:19.918 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 386 16 1935 1919 0.31068 0 - 0.363217 2024-04-16T07:55:20.918 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 387 16 1935 1919 0.309877 0 - 0.363217 2024-04-16T07:55:21.918 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 388 16 1935 1919 0.309079 0 - 0.363217 2024-04-16T07:55:22.919 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 389 16 1935 1919 0.308284 0 - 0.363217 2024-04-16T07:55:23.919 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 390 16 1935 1919 0.307493 0 - 0.363217 2024-04-16T07:55:24.919 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 391 16 1935 1919 0.306707 0 - 0.363217 2024-04-16T07:55:25.919 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 392 16 1935 1919 0.305925 0 - 0.363217 2024-04-16T07:55:26.919 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 393 16 1935 1919 0.305146 0 - 0.363217 2024-04-16T07:55:27.919 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 394 16 1935 1919 0.304372 0 - 0.363217 2024-04-16T07:55:28.919 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 395 16 1935 1919 0.303601 0 - 0.363217 2024-04-16T07:55:29.919 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 396 16 1935 1919 0.302835 0 - 0.363217 2024-04-16T07:55:30.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 397 16 1935 1919 0.302072 0 - 0.363217 2024-04-16T07:55:31.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 398 16 1935 1919 0.301313 0 - 0.363217 2024-04-16T07:55:32.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 399 16 1935 1919 0.300558 0 - 0.363217 2024-04-16T07:55:33.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:55:33.921347+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:55:33.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:55:33.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 400 16 1935 1919 0.299806 0 - 0.363217 2024-04-16T07:55:34.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 401 16 1935 1919 0.299059 0 - 0.363217 2024-04-16T07:55:35.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 402 16 1935 1919 0.298315 0 - 0.363217 2024-04-16T07:55:36.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 403 16 1935 1919 0.297574 0 - 0.363217 2024-04-16T07:55:37.920 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 404 16 1935 1919 0.296838 0 - 0.363217 2024-04-16T07:55:38.921 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 405 16 1935 1919 0.296105 0 - 0.363217 2024-04-16T07:55:39.921 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 406 16 1935 1919 0.295375 0 - 0.363217 2024-04-16T07:55:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T07:55:40.871 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:55:40.921 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 407 16 1935 1919 0.29465 0 - 0.363217 2024-04-16T07:55:41.921 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 408 16 1935 1919 0.293928 0 - 0.363217 2024-04-16T07:55:42.872 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:55:42.921 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 409 16 1935 1919 0.293209 0 - 0.363217 2024-04-16T07:55:42.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:55:42.993+0000 7ff928a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:55:43.921 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 410 16 1935 1919 0.292494 0 - 0.363217 2024-04-16T07:55:44.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 411 16 1935 1919 0.291782 0 - 0.363217 2024-04-16T07:55:45.922 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 412 16 1935 1919 0.291074 0 - 0.363217 2024-04-16T07:55:46.922 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 413 16 1935 1919 0.290369 0 - 0.363217 2024-04-16T07:55:47.922 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 414 16 1935 1919 0.289668 0 - 0.363217 2024-04-16T07:55:48.922 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 415 16 1935 1919 0.28897 0 - 0.363217 2024-04-16T07:55:49.922 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 416 16 1935 1919 0.288275 0 - 0.363217 2024-04-16T07:55:50.922 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 417 16 1935 1919 0.287584 0 - 0.363217 2024-04-16T07:55:51.922 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 418 16 1935 1919 0.286896 0 - 0.363217 2024-04-16T07:55:52.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 419 16 1935 1919 0.286211 0 - 0.363217 2024-04-16T07:55:53.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:55:53.924058+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:55:53.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:55:53.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 420 16 1935 1919 0.28553 0 - 0.363217 2024-04-16T07:55:54.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 421 16 1935 1919 0.284851 0 - 0.363217 2024-04-16T07:55:55.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 422 16 1935 1919 0.284176 0 - 0.363217 2024-04-16T07:55:56.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 423 16 1935 1919 0.283504 0 - 0.363217 2024-04-16T07:55:57.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 424 16 1935 1919 0.282836 0 - 0.363217 2024-04-16T07:55:58.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 425 16 1935 1919 0.28217 0 - 0.363217 2024-04-16T07:55:59.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 426 16 1935 1919 0.281508 0 - 0.363217 2024-04-16T07:56:00.923 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 427 16 1935 1919 0.280849 0 - 0.363217 2024-04-16T07:56:01.924 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 428 16 1935 1919 0.280193 0 - 0.363217 2024-04-16T07:56:02.924 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 429 16 1935 1919 0.279539 0 - 0.363217 2024-04-16T07:56:03.924 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 430 16 1935 1919 0.278889 0 - 0.363217 2024-04-16T07:56:04.924 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 431 16 1935 1919 0.278242 0 - 0.363217 2024-04-16T07:56:05.924 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 432 16 1935 1919 0.277598 0 - 0.363217 2024-04-16T07:56:06.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 433 16 1935 1919 0.276957 0 - 0.363217 2024-04-16T07:56:07.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 434 16 1935 1919 0.276319 0 - 0.363217 2024-04-16T07:56:08.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 435 16 1935 1919 0.275684 0 - 0.363217 2024-04-16T07:56:09.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 436 16 1935 1919 0.275051 0 - 0.363217 2024-04-16T07:56:10.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 437 16 1935 1919 0.274422 0 - 0.363217 2024-04-16T07:56:11.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 438 16 1935 1919 0.273795 0 - 0.363217 2024-04-16T07:56:12.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 439 16 1935 1919 0.273172 0 - 0.363217 2024-04-16T07:56:13.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:56:13.926691+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:56:13.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:56:13.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 440 16 1935 1919 0.272551 0 - 0.363217 2024-04-16T07:56:14.925 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 441 16 1935 1919 0.271933 0 - 0.363217 2024-04-16T07:56:15.926 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 442 16 1935 1919 0.271318 0 - 0.363217 2024-04-16T07:56:16.926 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 443 16 1935 1919 0.270705 0 - 0.363217 2024-04-16T07:56:17.926 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 444 16 1935 1919 0.270095 0 - 0.363217 2024-04-16T07:56:18.926 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 445 16 1935 1919 0.269488 0 - 0.363217 2024-04-16T07:56:19.926 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 446 16 1935 1919 0.268884 0 - 0.363217 2024-04-16T07:56:20.926 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 447 16 1935 1919 0.268283 0 - 0.363217 2024-04-16T07:56:21.926 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 448 16 1935 1919 0.267684 0 - 0.363217 2024-04-16T07:56:22.926 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 449 16 1935 1919 0.267088 0 - 0.363217 2024-04-16T07:56:23.927 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 450 16 1935 1919 0.266494 0 - 0.363217 2024-04-16T07:56:24.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 451 16 1935 1919 0.265903 0 - 0.363217 2024-04-16T07:56:25.927 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 452 16 1935 1919 0.265315 0 - 0.363217 2024-04-16T07:56:26.927 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 453 16 1935 1919 0.264729 0 - 0.363217 2024-04-16T07:56:27.927 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 454 16 1935 1919 0.264146 0 - 0.363217 2024-04-16T07:56:28.927 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 455 16 1935 1919 0.263566 0 - 0.363217 2024-04-16T07:56:29.927 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 456 16 1935 1919 0.262988 0 - 0.363217 2024-04-16T07:56:30.927 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 457 16 1935 1919 0.262412 0 - 0.363217 2024-04-16T07:56:31.928 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 458 16 1935 1919 0.261839 0 - 0.363217 2024-04-16T07:56:32.928 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 459 16 1935 1919 0.261269 0 - 0.363217 2024-04-16T07:56:33.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:56:33.929368+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:56:33.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:56:33.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 460 16 1935 1919 0.260701 0 - 0.363217 2024-04-16T07:56:34.928 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 461 16 1935 1919 0.260135 0 - 0.363217 2024-04-16T07:56:35.928 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 462 16 1935 1919 0.259572 0 - 0.363217 2024-04-16T07:56:36.928 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 463 16 1935 1919 0.259011 0 - 0.363217 2024-04-16T07:56:37.928 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 464 16 1935 1919 0.258453 0 - 0.363217 2024-04-16T07:56:38.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 465 16 1935 1919 0.257897 0 - 0.363217 2024-04-16T07:56:39.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 466 16 1935 1919 0.257344 0 - 0.363217 2024-04-16T07:56:40.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 467 16 1935 1919 0.256793 0 - 0.363217 2024-04-16T07:56:41.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 468 16 1935 1919 0.256244 0 - 0.363217 2024-04-16T07:56:42.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 469 16 1935 1919 0.255698 0 - 0.363217 2024-04-16T07:56:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T07:56:43.302 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1453, in _do_thrash self.ceph_manager.wait_for_recovery( File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2905, in wait_for_recovery while not self.is_recovered(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2770, in is_recovered pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T07:56:43.302 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1332, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1453, in _do_thrash self.ceph_manager.wait_for_recovery( File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2905, in wait_for_recovery while not self.is_recovered(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2770, in is_recovered pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T07:56:43.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 470 16 1935 1919 0.255154 0 - 0.363217 2024-04-16T07:56:44.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 471 16 1935 1919 0.254612 0 - 0.363217 2024-04-16T07:56:45.929 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 472 16 1935 1919 0.254073 0 - 0.363217 2024-04-16T07:56:46.930 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 473 16 1935 1919 0.253536 0 - 0.363217 2024-04-16T07:56:47.930 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 474 16 1935 1919 0.253001 0 - 0.363217 2024-04-16T07:56:48.930 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 475 16 1935 1919 0.252468 0 - 0.363217 2024-04-16T07:56:49.930 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 476 16 1935 1919 0.251938 0 - 0.363217 2024-04-16T07:56:50.930 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 477 16 1935 1919 0.251409 0 - 0.363217 2024-04-16T07:56:51.930 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 478 16 1935 1919 0.250883 0 - 0.363217 2024-04-16T07:56:52.930 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 479 16 1935 1919 0.25036 0 - 0.363217 2024-04-16T07:56:53.931 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:56:53.931958+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:56:53.931 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:56:53.931 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 480 16 1935 1919 0.249838 0 - 0.363217 2024-04-16T07:56:54.931 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 481 16 1935 1919 0.249319 0 - 0.363217 2024-04-16T07:56:55.931 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 482 16 1935 1919 0.248801 0 - 0.363217 2024-04-16T07:56:56.931 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 483 16 1935 1919 0.248286 0 - 0.363217 2024-04-16T07:56:57.931 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 484 16 1935 1919 0.247773 0 - 0.363217 2024-04-16T07:56:58.931 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 485 16 1935 1919 0.247263 0 - 0.363217 2024-04-16T07:56:59.931 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 486 16 1935 1919 0.246754 0 - 0.363217 2024-04-16T07:57:00.932 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 487 16 1935 1919 0.246247 0 - 0.363217 2024-04-16T07:57:01.932 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 488 16 1935 1919 0.245742 0 - 0.363217 2024-04-16T07:57:02.932 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 489 16 1935 1919 0.24524 0 - 0.363217 2024-04-16T07:57:03.932 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 490 16 1935 1919 0.244739 0 - 0.363217 2024-04-16T07:57:04.932 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 491 16 1935 1919 0.244241 0 - 0.363217 2024-04-16T07:57:05.932 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 492 16 1935 1919 0.243745 0 - 0.363217 2024-04-16T07:57:06.932 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 493 16 1935 1919 0.24325 0 - 0.363217 2024-04-16T07:57:07.932 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 494 16 1935 1919 0.242758 0 - 0.363217 2024-04-16T07:57:08.932 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 495 16 1935 1919 0.242267 0 - 0.363217 2024-04-16T07:57:09.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 496 16 1935 1919 0.241779 0 - 0.363217 2024-04-16T07:57:10.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 497 16 1935 1919 0.241292 0 - 0.363217 2024-04-16T07:57:11.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 498 16 1935 1919 0.240808 0 - 0.363217 2024-04-16T07:57:12.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 499 16 1935 1919 0.240325 0 - 0.363217 2024-04-16T07:57:13.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:57:13.934587+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:57:13.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:57:13.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 500 16 1935 1919 0.239845 0 - 0.363217 2024-04-16T07:57:14.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 501 16 1935 1919 0.239366 0 - 0.363217 2024-04-16T07:57:15.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 502 16 1935 1919 0.238889 0 - 0.363217 2024-04-16T07:57:16.933 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 503 16 1935 1919 0.238414 0 - 0.363217 2024-04-16T07:57:17.934 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 504 16 1935 1919 0.237941 0 - 0.363217 2024-04-16T07:57:18.934 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 505 16 1935 1919 0.23747 0 - 0.363217 2024-04-16T07:57:19.934 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 506 16 1935 1919 0.237001 0 - 0.363217 2024-04-16T07:57:20.934 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 507 16 1935 1919 0.236533 0 - 0.363217 2024-04-16T07:57:21.934 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 508 16 1935 1919 0.236067 0 - 0.363217 2024-04-16T07:57:22.935 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 509 16 1935 1919 0.235604 0 - 0.363217 2024-04-16T07:57:23.938 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 510 16 1935 1919 0.235142 0 - 0.363217 2024-04-16T07:57:24.935 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 511 16 1935 1919 0.234682 0 - 0.363217 2024-04-16T07:57:25.935 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 512 16 1935 1919 0.234223 0 - 0.363217 2024-04-16T07:57:26.935 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 513 16 1935 1919 0.233767 0 - 0.363217 2024-04-16T07:57:27.935 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 514 16 1935 1919 0.233312 0 - 0.363217 2024-04-16T07:57:28.935 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 515 16 1935 1919 0.232859 0 - 0.363217 2024-04-16T07:57:29.935 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 516 16 1935 1919 0.232407 0 - 0.363217 2024-04-16T07:57:30.935 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 517 16 1935 1919 0.231958 0 - 0.363217 2024-04-16T07:57:31.936 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 518 16 1935 1919 0.23151 0 - 0.363217 2024-04-16T07:57:32.936 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 519 16 1935 1919 0.231064 0 - 0.363217 2024-04-16T07:57:33.936 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:57:33.937349+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:57:33.936 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:57:33.936 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 520 16 1935 1919 0.23062 0 - 0.363217 2024-04-16T07:57:34.936 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 521 16 1935 1919 0.230177 0 - 0.363217 2024-04-16T07:57:35.936 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 522 16 1935 1919 0.229736 0 - 0.363217 2024-04-16T07:57:36.936 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 523 16 1935 1919 0.229297 0 - 0.363217 2024-04-16T07:57:37.936 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 524 16 1935 1919 0.228859 0 - 0.363217 2024-04-16T07:57:38.937 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 525 16 1935 1919 0.228423 0 - 0.363217 2024-04-16T07:57:39.937 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 526 16 1935 1919 0.227989 0 - 0.363217 2024-04-16T07:57:40.937 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 527 16 1935 1919 0.227556 0 - 0.363217 2024-04-16T07:57:41.937 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 528 16 1935 1919 0.227125 0 - 0.363217 2024-04-16T07:57:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T07:57:42.912 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:57:42.937 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 529 16 1935 1919 0.226696 0 - 0.363217 2024-04-16T07:57:43.937 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 530 16 1935 1919 0.226268 0 - 0.363217 2024-04-16T07:57:44.912 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T07:57:44.937 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 531 16 1935 1919 0.225842 0 - 0.363217 2024-04-16T07:57:45.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:57:45.033+0000 7f72a1813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:57:45.937 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 532 16 1935 1919 0.225418 0 - 0.363217 2024-04-16T07:57:46.937 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 533 16 1935 1919 0.224995 0 - 0.363217 2024-04-16T07:57:47.938 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 534 16 1935 1919 0.224573 0 - 0.363217 2024-04-16T07:57:48.938 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 535 16 1935 1919 0.224154 0 - 0.363217 2024-04-16T07:57:49.938 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 536 16 1935 1919 0.223736 0 - 0.363217 2024-04-16T07:57:50.938 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 537 16 1935 1919 0.223319 0 - 0.363217 2024-04-16T07:57:51.938 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 538 16 1935 1919 0.222904 0 - 0.363217 2024-04-16T07:57:52.938 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 539 16 1935 1919 0.22249 0 - 0.363217 2024-04-16T07:57:53.938 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:57:53.939942+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:57:53.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:57:53.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 540 16 1935 1919 0.222078 0 - 0.363217 2024-04-16T07:57:54.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 541 16 1935 1919 0.221668 0 - 0.363217 2024-04-16T07:57:55.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 542 16 1935 1919 0.221259 0 - 0.363217 2024-04-16T07:57:56.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 543 16 1935 1919 0.220851 0 - 0.363217 2024-04-16T07:57:57.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 544 16 1935 1919 0.220445 0 - 0.363217 2024-04-16T07:57:58.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 545 16 1935 1919 0.220041 0 - 0.363217 2024-04-16T07:57:59.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 546 16 1935 1919 0.219638 0 - 0.363217 2024-04-16T07:58:00.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 547 16 1935 1919 0.219236 0 - 0.363217 2024-04-16T07:58:01.939 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 548 16 1935 1919 0.218836 0 - 0.363217 2024-04-16T07:58:02.940 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 549 16 1935 1919 0.218438 0 - 0.363217 2024-04-16T07:58:03.940 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 550 16 1935 1919 0.21804 0 - 0.363217 2024-04-16T07:58:04.940 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 551 16 1935 1919 0.217645 0 - 0.363217 2024-04-16T07:58:05.940 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 552 16 1935 1919 0.21725 0 - 0.363217 2024-04-16T07:58:06.940 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 553 16 1935 1919 0.216858 0 - 0.363217 2024-04-16T07:58:07.940 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 554 16 1935 1919 0.216466 0 - 0.363217 2024-04-16T07:58:08.941 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 555 16 1935 1919 0.216076 0 - 0.363217 2024-04-16T07:58:09.940 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 556 16 1935 1919 0.215687 0 - 0.363217 2024-04-16T07:58:10.941 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 557 16 1935 1919 0.2153 0 - 0.363217 2024-04-16T07:58:11.941 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 558 16 1935 1919 0.214914 0 - 0.363217 2024-04-16T07:58:12.941 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 559 16 1935 1919 0.21453 0 - 0.363217 2024-04-16T07:58:13.941 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:58:13.942618+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:58:13.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:58:13.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 560 16 1935 1919 0.214147 0 - 0.363217 2024-04-16T07:58:14.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 561 16 1935 1919 0.213765 0 - 0.363217 2024-04-16T07:58:15.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 562 16 1935 1919 0.213385 0 - 0.363217 2024-04-16T07:58:16.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 563 16 1935 1919 0.213006 0 - 0.363217 2024-04-16T07:58:17.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 564 16 1935 1919 0.212628 0 - 0.363217 2024-04-16T07:58:18.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 565 16 1935 1919 0.212252 0 - 0.363217 2024-04-16T07:58:19.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 566 16 1935 1919 0.211877 0 - 0.363217 2024-04-16T07:58:20.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 567 16 1935 1919 0.211503 0 - 0.363217 2024-04-16T07:58:21.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 568 16 1935 1919 0.211131 0 - 0.363217 2024-04-16T07:58:22.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 569 16 1935 1919 0.21076 0 - 0.363217 2024-04-16T07:58:23.942 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 570 16 1935 1919 0.21039 0 - 0.363217 2024-04-16T07:58:24.943 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 571 16 1935 1919 0.210021 0 - 0.363217 2024-04-16T07:58:25.943 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 572 16 1935 1919 0.209654 0 - 0.363217 2024-04-16T07:58:26.943 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 573 16 1935 1919 0.209288 0 - 0.363217 2024-04-16T07:58:27.943 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 574 16 1935 1919 0.208924 0 - 0.363217 2024-04-16T07:58:28.943 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 575 16 1935 1919 0.20856 0 - 0.363217 2024-04-16T07:58:29.943 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 576 16 1935 1919 0.208198 0 - 0.363217 2024-04-16T07:58:30.943 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 577 16 1935 1919 0.207837 0 - 0.363217 2024-04-16T07:58:31.943 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 578 16 1935 1919 0.207478 0 - 0.363217 2024-04-16T07:58:32.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 579 16 1935 1919 0.207119 0 - 0.363217 2024-04-16T07:58:33.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:58:33.945270+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:58:33.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:58:33.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 580 16 1935 1919 0.206762 0 - 0.363217 2024-04-16T07:58:34.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 581 16 1935 1919 0.206407 0 - 0.363217 2024-04-16T07:58:35.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 582 16 1935 1919 0.206052 0 - 0.363217 2024-04-16T07:58:36.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 583 16 1935 1919 0.205698 0 - 0.363217 2024-04-16T07:58:37.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 584 16 1935 1919 0.205346 0 - 0.363217 2024-04-16T07:58:38.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 585 16 1935 1919 0.204995 0 - 0.363217 2024-04-16T07:58:39.944 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 586 16 1935 1919 0.204645 0 - 0.363217 2024-04-16T07:58:40.945 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 587 16 1935 1919 0.204297 0 - 0.363217 2024-04-16T07:58:41.945 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 588 16 1935 1919 0.203949 0 - 0.363217 2024-04-16T07:58:42.945 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 589 16 1935 1919 0.203603 0 - 0.363217 2024-04-16T07:58:43.945 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 590 16 1935 1919 0.203258 0 - 0.363217 2024-04-16T07:58:44.945 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 591 16 1935 1919 0.202914 0 - 0.363217 2024-04-16T07:58:45.945 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 592 16 1935 1919 0.202571 0 - 0.363217 2024-04-16T07:58:46.945 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 593 16 1935 1919 0.20223 0 - 0.363217 2024-04-16T07:58:47.945 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 594 16 1935 1919 0.201889 0 - 0.363217 2024-04-16T07:58:48.946 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 595 16 1935 1919 0.20155 0 - 0.363217 2024-04-16T07:58:49.946 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 596 16 1935 1919 0.201212 0 - 0.363217 2024-04-16T07:58:50.946 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 597 16 1935 1919 0.200875 0 - 0.363217 2024-04-16T07:58:51.946 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 598 16 1935 1919 0.200539 0 - 0.363217 2024-04-16T07:58:52.946 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 599 16 1935 1919 0.200204 0 - 0.363217 2024-04-16T07:58:53.946 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:58:53.947850+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:58:53.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:58:53.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 600 16 1935 1919 0.19987 0 - 0.363217 2024-04-16T07:58:54.946 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 601 16 1935 1919 0.199538 0 - 0.363217 2024-04-16T07:58:55.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 602 16 1935 1919 0.199206 0 - 0.363217 2024-04-16T07:58:56.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 603 16 1935 1919 0.198876 0 - 0.363217 2024-04-16T07:58:57.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 604 16 1935 1919 0.198547 0 - 0.363217 2024-04-16T07:58:58.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 605 16 1935 1919 0.198218 0 - 0.363217 2024-04-16T07:58:59.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 606 16 1935 1919 0.197891 0 - 0.363217 2024-04-16T07:59:00.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 607 16 1935 1919 0.197565 0 - 0.363217 2024-04-16T07:59:01.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 608 16 1935 1919 0.19724 0 - 0.363217 2024-04-16T07:59:02.948 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 609 16 1935 1919 0.196917 0 - 0.363217 2024-04-16T07:59:03.947 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 610 16 1935 1919 0.196594 0 - 0.363217 2024-04-16T07:59:04.948 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 611 16 1935 1919 0.196272 0 - 0.363217 2024-04-16T07:59:05.948 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 612 16 1935 1919 0.195951 0 - 0.363217 2024-04-16T07:59:06.948 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 613 16 1935 1919 0.195632 0 - 0.363217 2024-04-16T07:59:07.948 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 614 16 1935 1919 0.195313 0 - 0.363217 2024-04-16T07:59:08.948 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 615 16 1935 1919 0.194995 0 - 0.363217 2024-04-16T07:59:09.948 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 616 16 1935 1919 0.194679 0 - 0.363217 2024-04-16T07:59:10.949 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 617 16 1935 1919 0.194363 0 - 0.363217 2024-04-16T07:59:11.949 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 618 16 1935 1919 0.194049 0 - 0.363217 2024-04-16T07:59:12.949 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 619 16 1935 1919 0.193735 0 - 0.363217 2024-04-16T07:59:13.949 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:59:13.950542+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:59:13.949 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:59:13.949 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 620 16 1935 1919 0.193423 0 - 0.363217 2024-04-16T07:59:14.949 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 621 16 1935 1919 0.193111 0 - 0.363217 2024-04-16T07:59:15.949 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 622 16 1935 1919 0.192801 0 - 0.363217 2024-04-16T07:59:16.949 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 623 16 1935 1919 0.192491 0 - 0.363217 2024-04-16T07:59:17.950 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 624 16 1935 1919 0.192183 0 - 0.363217 2024-04-16T07:59:18.950 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 625 16 1935 1919 0.191875 0 - 0.363217 2024-04-16T07:59:19.950 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 626 16 1935 1919 0.191569 0 - 0.363217 2024-04-16T07:59:20.950 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 627 16 1935 1919 0.191263 0 - 0.363217 2024-04-16T07:59:21.950 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 628 16 1935 1919 0.190959 0 - 0.363217 2024-04-16T07:59:22.950 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 629 16 1935 1919 0.190655 0 - 0.363217 2024-04-16T07:59:23.950 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 630 16 1935 1919 0.190353 0 - 0.363217 2024-04-16T07:59:24.950 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 631 16 1935 1919 0.190051 0 - 0.363217 2024-04-16T07:59:25.951 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 632 16 1935 1919 0.18975 0 - 0.363217 2024-04-16T07:59:26.951 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 633 16 1935 1919 0.18945 0 - 0.363217 2024-04-16T07:59:27.951 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 634 16 1935 1919 0.189152 0 - 0.363217 2024-04-16T07:59:28.951 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 635 16 1935 1919 0.188854 0 - 0.363217 2024-04-16T07:59:29.951 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 636 16 1935 1919 0.188557 0 - 0.363217 2024-04-16T07:59:30.951 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 637 16 1935 1919 0.188261 0 - 0.363217 2024-04-16T07:59:31.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 638 16 1935 1919 0.187966 0 - 0.363217 2024-04-16T07:59:32.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 639 16 1935 1919 0.187672 0 - 0.363217 2024-04-16T07:59:33.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:59:33.953312+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:59:33.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:59:33.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 640 16 1935 1919 0.187378 0 - 0.363217 2024-04-16T07:59:34.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 641 16 1935 1919 0.187086 0 - 0.363217 2024-04-16T07:59:35.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 642 16 1935 1919 0.186795 0 - 0.363217 2024-04-16T07:59:36.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 643 16 1935 1919 0.186504 0 - 0.363217 2024-04-16T07:59:37.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 644 16 1935 1919 0.186214 0 - 0.363217 2024-04-16T07:59:38.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 645 16 1935 1919 0.185926 0 - 0.363217 2024-04-16T07:59:39.952 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 646 16 1935 1919 0.185638 0 - 0.363217 2024-04-16T07:59:40.953 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 647 16 1935 1919 0.185351 0 - 0.363217 2024-04-16T07:59:41.953 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 648 16 1935 1919 0.185065 0 - 0.363217 2024-04-16T07:59:42.953 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 649 16 1935 1919 0.18478 0 - 0.363217 2024-04-16T07:59:43.953 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 650 16 1935 1919 0.184496 0 - 0.363217 2024-04-16T07:59:44.952 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T07:59:44.953 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T07:59:44.953 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 651 16 1935 1919 0.184212 0 - 0.363217 2024-04-16T07:59:45.953 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 652 16 1935 1919 0.18393 0 - 0.363217 2024-04-16T07:59:46.953 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T07:59:46.955 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 653 16 1935 1919 0.183648 0 - 0.363217 2024-04-16T07:59:47.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T07:59:47.074+0000 7f3c80864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:59:47.954 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 654 16 1935 1919 0.183367 0 - 0.363217 2024-04-16T07:59:48.954 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 655 16 1935 1919 0.183087 0 - 0.363217 2024-04-16T07:59:49.954 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 656 16 1935 1919 0.182808 0 - 0.363217 2024-04-16T07:59:50.954 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 657 16 1935 1919 0.18253 0 - 0.363217 2024-04-16T07:59:51.954 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 658 16 1935 1919 0.182252 0 - 0.363217 2024-04-16T07:59:52.954 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 659 16 1935 1919 0.181976 0 - 0.363217 2024-04-16T07:59:53.954 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T07:59:53.955949+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T07:59:53.954 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T07:59:53.955 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 660 16 1935 1919 0.1817 0 - 0.363217 2024-04-16T07:59:54.954 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 661 16 1935 1919 0.181425 0 - 0.363217 2024-04-16T07:59:55.955 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 662 16 1935 1919 0.181151 0 - 0.363217 2024-04-16T07:59:56.955 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 663 16 1935 1919 0.180878 0 - 0.363217 2024-04-16T07:59:57.955 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 664 16 1935 1919 0.180606 0 - 0.363217 2024-04-16T07:59:58.955 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 665 16 1935 1919 0.180334 0 - 0.363217 2024-04-16T07:59:59.955 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 666 16 1935 1919 0.180063 0 - 0.363217 2024-04-16T08:00:00.956 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 667 16 1935 1919 0.179793 0 - 0.363217 2024-04-16T08:00:01.955 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 668 16 1935 1919 0.179524 0 - 0.363217 2024-04-16T08:00:02.955 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 669 16 1935 1919 0.179256 0 - 0.363217 2024-04-16T08:00:03.956 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 670 16 1935 1919 0.178988 0 - 0.363217 2024-04-16T08:00:04.956 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 671 16 1935 1919 0.178721 0 - 0.363217 2024-04-16T08:00:05.956 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 672 16 1935 1919 0.178455 0 - 0.363217 2024-04-16T08:00:06.956 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 673 16 1935 1919 0.17819 0 - 0.363217 2024-04-16T08:00:07.956 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 674 16 1935 1919 0.177926 0 - 0.363217 2024-04-16T08:00:08.956 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 675 16 1935 1919 0.177662 0 - 0.363217 2024-04-16T08:00:09.956 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 676 16 1935 1919 0.1774 0 - 0.363217 2024-04-16T08:00:10.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 677 16 1935 1919 0.177137 0 - 0.363217 2024-04-16T08:00:11.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 678 16 1935 1919 0.176876 0 - 0.363217 2024-04-16T08:00:12.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 679 16 1935 1919 0.176616 0 - 0.363217 2024-04-16T08:00:13.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:00:13.958664+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:00:13.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:00:13.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 680 16 1935 1919 0.176356 0 - 0.363217 2024-04-16T08:00:14.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 681 16 1935 1919 0.176097 0 - 0.363217 2024-04-16T08:00:15.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 682 16 1935 1919 0.175839 0 - 0.363217 2024-04-16T08:00:16.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 683 16 1935 1919 0.175581 0 - 0.363217 2024-04-16T08:00:17.957 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 684 16 1935 1919 0.175325 0 - 0.363217 2024-04-16T08:00:18.958 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 685 16 1935 1919 0.175069 0 - 0.363217 2024-04-16T08:00:19.958 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 686 16 1935 1919 0.174814 0 - 0.363217 2024-04-16T08:00:20.958 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 687 16 1935 1919 0.174559 0 - 0.363217 2024-04-16T08:00:21.958 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 688 16 1935 1919 0.174305 0 - 0.363217 2024-04-16T08:00:22.958 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 689 16 1935 1919 0.174052 0 - 0.363217 2024-04-16T08:00:23.958 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 690 16 1935 1919 0.1738 0 - 0.363217 2024-04-16T08:00:24.959 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 691 16 1935 1919 0.173549 0 - 0.363217 2024-04-16T08:00:25.959 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 692 16 1935 1919 0.173298 0 - 0.363217 2024-04-16T08:00:26.959 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 693 16 1935 1919 0.173048 0 - 0.363217 2024-04-16T08:00:27.959 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 694 16 1935 1919 0.172798 0 - 0.363217 2024-04-16T08:00:28.959 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 695 16 1935 1919 0.17255 0 - 0.363217 2024-04-16T08:00:29.959 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 696 16 1935 1919 0.172302 0 - 0.363217 2024-04-16T08:00:30.959 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 697 16 1935 1919 0.172055 0 - 0.363217 2024-04-16T08:00:31.959 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 698 16 1935 1919 0.171808 0 - 0.363217 2024-04-16T08:00:32.960 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 699 16 1935 1919 0.171562 0 - 0.363217 2024-04-16T08:00:33.960 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:00:33.961378+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:00:33.960 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:00:33.960 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 700 16 1935 1919 0.171317 0 - 0.363217 2024-04-16T08:00:34.960 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 701 16 1935 1919 0.171073 0 - 0.363217 2024-04-16T08:00:35.960 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 702 16 1935 1919 0.170829 0 - 0.363217 2024-04-16T08:00:36.960 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 703 16 1935 1919 0.170586 0 - 0.363217 2024-04-16T08:00:37.960 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 704 16 1935 1919 0.170344 0 - 0.363217 2024-04-16T08:00:38.960 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 705 16 1935 1919 0.170102 0 - 0.363217 2024-04-16T08:00:39.961 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 706 16 1935 1919 0.169861 0 - 0.363217 2024-04-16T08:00:40.961 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 707 16 1935 1919 0.169621 0 - 0.363217 2024-04-16T08:00:41.961 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 708 16 1935 1919 0.169381 0 - 0.363217 2024-04-16T08:00:42.961 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 709 16 1935 1919 0.169143 0 - 0.363217 2024-04-16T08:00:43.961 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 710 16 1935 1919 0.168904 0 - 0.363217 2024-04-16T08:00:44.961 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 711 16 1935 1919 0.168667 0 - 0.363217 2024-04-16T08:00:45.961 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 712 16 1935 1919 0.16843 0 - 0.363217 2024-04-16T08:00:46.961 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 713 16 1935 1919 0.168194 0 - 0.363217 2024-04-16T08:00:47.962 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 714 16 1935 1919 0.167958 0 - 0.363217 2024-04-16T08:00:48.962 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 715 16 1935 1919 0.167723 0 - 0.363217 2024-04-16T08:00:49.962 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 716 16 1935 1919 0.167489 0 - 0.363217 2024-04-16T08:00:50.962 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 717 16 1935 1919 0.167255 0 - 0.363217 2024-04-16T08:00:51.962 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 718 16 1935 1919 0.167022 0 - 0.363217 2024-04-16T08:00:52.962 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 719 16 1935 1919 0.16679 0 - 0.363217 2024-04-16T08:00:53.962 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:00:53.964072+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:00:53.963 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:00:53.963 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 720 16 1935 1919 0.166558 0 - 0.363217 2024-04-16T08:00:54.963 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 721 16 1935 1919 0.166327 0 - 0.363217 2024-04-16T08:00:55.963 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 722 16 1935 1919 0.166097 0 - 0.363217 2024-04-16T08:00:56.963 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 723 16 1935 1919 0.165867 0 - 0.363217 2024-04-16T08:00:57.963 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 724 16 1935 1919 0.165638 0 - 0.363217 2024-04-16T08:00:58.963 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 725 16 1935 1919 0.16541 0 - 0.363217 2024-04-16T08:00:59.963 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 726 16 1935 1919 0.165182 0 - 0.363217 2024-04-16T08:01:00.963 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 727 16 1935 1919 0.164955 0 - 0.363217 2024-04-16T08:01:01.964 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 728 16 1935 1919 0.164728 0 - 0.363217 2024-04-16T08:01:02.964 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 729 16 1935 1919 0.164502 0 - 0.363217 2024-04-16T08:01:03.964 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 730 16 1935 1919 0.164277 0 - 0.363217 2024-04-16T08:01:04.964 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 731 16 1935 1919 0.164052 0 - 0.363217 2024-04-16T08:01:05.964 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 732 16 1935 1919 0.163828 0 - 0.363217 2024-04-16T08:01:06.964 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 733 16 1935 1919 0.163604 0 - 0.363217 2024-04-16T08:01:07.964 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 734 16 1935 1919 0.163381 0 - 0.363217 2024-04-16T08:01:08.965 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 735 16 1935 1919 0.163159 0 - 0.363217 2024-04-16T08:01:09.965 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 736 16 1935 1919 0.162938 0 - 0.363217 2024-04-16T08:01:10.965 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 737 16 1935 1919 0.162716 0 - 0.363217 2024-04-16T08:01:11.965 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 738 16 1935 1919 0.162496 0 - 0.363217 2024-04-16T08:01:12.965 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 739 16 1935 1919 0.162276 0 - 0.363217 2024-04-16T08:01:13.965 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:01:13.966893+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:01:13.965 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:01:13.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 740 16 1935 1919 0.162057 0 - 0.363217 2024-04-16T08:01:14.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 741 16 1935 1919 0.161838 0 - 0.363217 2024-04-16T08:01:15.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 742 16 1935 1919 0.16162 0 - 0.363217 2024-04-16T08:01:16.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 743 16 1935 1919 0.161402 0 - 0.363217 2024-04-16T08:01:17.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 744 16 1935 1919 0.161185 0 - 0.363217 2024-04-16T08:01:18.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 745 16 1935 1919 0.160969 0 - 0.363217 2024-04-16T08:01:19.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 746 16 1935 1919 0.160753 0 - 0.363217 2024-04-16T08:01:20.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 747 16 1935 1919 0.160538 0 - 0.363217 2024-04-16T08:01:21.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 748 16 1935 1919 0.160324 0 - 0.363217 2024-04-16T08:01:22.966 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 749 16 1935 1919 0.160109 0 - 0.363217 2024-04-16T08:01:23.967 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 750 16 1935 1919 0.159896 0 - 0.363217 2024-04-16T08:01:24.967 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 751 16 1935 1919 0.159683 0 - 0.363217 2024-04-16T08:01:25.967 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 752 16 1935 1919 0.159471 0 - 0.363217 2024-04-16T08:01:26.967 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 753 16 1935 1919 0.159259 0 - 0.363217 2024-04-16T08:01:27.967 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 754 16 1935 1919 0.159048 0 - 0.363217 2024-04-16T08:01:28.967 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 755 16 1935 1919 0.158837 0 - 0.363217 2024-04-16T08:01:29.967 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 756 16 1935 1919 0.158627 0 - 0.363217 2024-04-16T08:01:30.967 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 757 16 1935 1919 0.158417 0 - 0.363217 2024-04-16T08:01:31.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 758 16 1935 1919 0.158208 0 - 0.363217 2024-04-16T08:01:32.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 759 16 1935 1919 0.158 0 - 0.363217 2024-04-16T08:01:33.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:01:33.969488+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:01:33.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:01:33.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 760 16 1935 1919 0.157792 0 - 0.363217 2024-04-16T08:01:34.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 761 16 1935 1919 0.157585 0 - 0.363217 2024-04-16T08:01:35.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 762 16 1935 1919 0.157378 0 - 0.363217 2024-04-16T08:01:36.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 763 16 1935 1919 0.157172 0 - 0.363217 2024-04-16T08:01:37.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 764 16 1935 1919 0.156966 0 - 0.363217 2024-04-16T08:01:38.968 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 765 16 1935 1919 0.156761 0 - 0.363217 2024-04-16T08:01:39.969 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 766 16 1935 1919 0.156556 0 - 0.363217 2024-04-16T08:01:40.969 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 767 16 1935 1919 0.156352 0 - 0.363217 2024-04-16T08:01:41.969 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 768 16 1935 1919 0.156148 0 - 0.363217 2024-04-16T08:01:42.969 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 769 16 1935 1919 0.155945 0 - 0.363217 2024-04-16T08:01:43.969 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 770 16 1935 1919 0.155743 0 - 0.363217 2024-04-16T08:01:44.969 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 771 16 1935 1919 0.155541 0 - 0.363217 2024-04-16T08:01:45.969 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 772 16 1935 1919 0.155339 0 - 0.363217 2024-04-16T08:01:46.970 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 773 16 1935 1919 0.155138 0 - 0.363217 2024-04-16T08:01:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:01:46.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:01:47.970 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 774 16 1935 1919 0.154938 0 - 0.363217 2024-04-16T08:01:48.970 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 775 16 1935 1919 0.154738 0 - 0.363217 2024-04-16T08:01:48.995 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:01:49.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:01:49.116+0000 7f3f1886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:01:49.970 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 776 16 1935 1919 0.154539 0 - 0.363217 2024-04-16T08:01:50.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 777 16 1935 1919 0.15434 0 - 0.363217 2024-04-16T08:01:51.970 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 778 16 1935 1919 0.154141 0 - 0.363217 2024-04-16T08:01:52.970 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 779 16 1935 1919 0.153943 0 - 0.363217 2024-04-16T08:01:53.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:01:53.972174+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:01:53.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:01:53.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 780 16 1935 1919 0.153746 0 - 0.363217 2024-04-16T08:01:54.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 781 16 1935 1919 0.153549 0 - 0.363217 2024-04-16T08:01:55.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 782 16 1935 1919 0.153353 0 - 0.363217 2024-04-16T08:01:56.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 783 16 1935 1919 0.153157 0 - 0.363217 2024-04-16T08:01:57.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 784 16 1935 1919 0.152962 0 - 0.363217 2024-04-16T08:01:58.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 785 16 1935 1919 0.152767 0 - 0.363217 2024-04-16T08:01:59.972 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 786 16 1935 1919 0.152572 0 - 0.363217 2024-04-16T08:02:00.971 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 787 16 1935 1919 0.152379 0 - 0.363217 2024-04-16T08:02:01.972 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 788 16 1935 1919 0.152185 0 - 0.363217 2024-04-16T08:02:02.972 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 789 16 1935 1919 0.151992 0 - 0.363217 2024-04-16T08:02:03.972 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 790 16 1935 1919 0.1518 0 - 0.363217 2024-04-16T08:02:04.972 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 791 16 1935 1919 0.151608 0 - 0.363217 2024-04-16T08:02:05.972 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 792 16 1935 1919 0.151417 0 - 0.363217 2024-04-16T08:02:06.972 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 793 16 1935 1919 0.151226 0 - 0.363217 2024-04-16T08:02:07.972 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 794 16 1935 1919 0.151035 0 - 0.363217 2024-04-16T08:02:08.972 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 795 16 1935 1919 0.150845 0 - 0.363217 2024-04-16T08:02:09.973 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 796 16 1935 1919 0.150656 0 - 0.363217 2024-04-16T08:02:10.973 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 797 16 1935 1919 0.150467 0 - 0.363217 2024-04-16T08:02:11.973 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 798 16 1935 1919 0.150278 0 - 0.363217 2024-04-16T08:02:12.973 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 799 16 1935 1919 0.15009 0 - 0.363217 2024-04-16T08:02:13.973 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:02:13.974816+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:02:13.973 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:02:13.974 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 800 16 1935 1919 0.149902 0 - 0.363217 2024-04-16T08:02:14.973 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 801 16 1935 1919 0.149715 0 - 0.363217 2024-04-16T08:02:15.973 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 802 16 1935 1919 0.149529 0 - 0.363217 2024-04-16T08:02:16.974 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 803 16 1935 1919 0.149342 0 - 0.363217 2024-04-16T08:02:17.974 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 804 16 1935 1919 0.149157 0 - 0.363217 2024-04-16T08:02:18.974 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 805 16 1935 1919 0.148971 0 - 0.363217 2024-04-16T08:02:19.974 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 806 16 1935 1919 0.148787 0 - 0.363217 2024-04-16T08:02:20.974 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 807 16 1935 1919 0.148602 0 - 0.363217 2024-04-16T08:02:21.974 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 808 16 1935 1919 0.148418 0 - 0.363217 2024-04-16T08:02:22.974 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 809 16 1935 1919 0.148235 0 - 0.363217 2024-04-16T08:02:23.974 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 810 16 1935 1919 0.148052 0 - 0.363217 2024-04-16T08:02:24.975 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 811 16 1935 1919 0.147869 0 - 0.363217 2024-04-16T08:02:25.975 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 812 16 1935 1919 0.147687 0 - 0.363217 2024-04-16T08:02:26.975 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 813 16 1935 1919 0.147505 0 - 0.363217 2024-04-16T08:02:27.975 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 814 16 1935 1919 0.147324 0 - 0.363217 2024-04-16T08:02:28.975 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 815 16 1935 1919 0.147144 0 - 0.363217 2024-04-16T08:02:29.975 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 816 16 1935 1919 0.146963 0 - 0.363217 2024-04-16T08:02:30.975 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 817 16 1935 1919 0.146783 0 - 0.363217 2024-04-16T08:02:31.976 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 818 16 1935 1919 0.146604 0 - 0.363217 2024-04-16T08:02:32.976 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 819 16 1935 1919 0.146425 0 - 0.363217 2024-04-16T08:02:33.976 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:02:33.977518+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:02:33.976 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:02:33.976 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 820 16 1935 1919 0.146246 0 - 0.363217 2024-04-16T08:02:34.976 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 821 16 1935 1919 0.146068 0 - 0.363217 2024-04-16T08:02:35.976 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 822 16 1935 1919 0.14589 0 - 0.363217 2024-04-16T08:02:36.976 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 823 16 1935 1919 0.145713 0 - 0.363217 2024-04-16T08:02:37.976 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 824 16 1935 1919 0.145536 0 - 0.363217 2024-04-16T08:02:38.977 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 825 16 1935 1919 0.14536 0 - 0.363217 2024-04-16T08:02:39.977 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 826 16 1935 1919 0.145184 0 - 0.363217 2024-04-16T08:02:40.977 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 827 16 1935 1919 0.145008 0 - 0.363217 2024-04-16T08:02:41.977 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 828 16 1935 1919 0.144833 0 - 0.363217 2024-04-16T08:02:42.977 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 829 16 1935 1919 0.144659 0 - 0.363217 2024-04-16T08:02:43.977 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 830 16 1935 1919 0.144484 0 - 0.363217 2024-04-16T08:02:44.977 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 831 16 1935 1919 0.14431 0 - 0.363217 2024-04-16T08:02:45.977 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 832 16 1935 1919 0.144137 0 - 0.363217 2024-04-16T08:02:46.978 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 833 16 1935 1919 0.143964 0 - 0.363217 2024-04-16T08:02:47.978 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 834 16 1935 1919 0.143791 0 - 0.363217 2024-04-16T08:02:48.978 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 835 16 1935 1919 0.143619 0 - 0.363217 2024-04-16T08:02:49.978 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 836 16 1935 1919 0.143447 0 - 0.363217 2024-04-16T08:02:50.978 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 837 16 1935 1919 0.143276 0 - 0.363217 2024-04-16T08:02:51.978 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 838 16 1935 1919 0.143105 0 - 0.363217 2024-04-16T08:02:52.980 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 839 16 1935 1919 0.142934 0 - 0.363217 2024-04-16T08:02:53.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:02:53.980116+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:02:53.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:02:53.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 840 16 1935 1919 0.142764 0 - 0.363217 2024-04-16T08:02:54.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 841 16 1935 1919 0.142594 0 - 0.363217 2024-04-16T08:02:55.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 842 16 1935 1919 0.142425 0 - 0.363217 2024-04-16T08:02:56.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 843 16 1935 1919 0.142256 0 - 0.363217 2024-04-16T08:02:57.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 844 16 1935 1919 0.142088 0 - 0.363217 2024-04-16T08:02:58.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 845 16 1935 1919 0.141919 0 - 0.363217 2024-04-16T08:02:59.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 846 16 1935 1919 0.141752 0 - 0.363217 2024-04-16T08:03:00.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 847 16 1935 1919 0.141584 0 - 0.363217 2024-04-16T08:03:01.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 848 16 1935 1919 0.141417 0 - 0.363217 2024-04-16T08:03:02.979 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 849 16 1935 1919 0.141251 0 - 0.363217 2024-04-16T08:03:03.980 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 850 16 1935 1919 0.141085 0 - 0.363217 2024-04-16T08:03:04.980 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 851 16 1935 1919 0.140919 0 - 0.363217 2024-04-16T08:03:05.980 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 852 16 1935 1919 0.140753 0 - 0.363217 2024-04-16T08:03:06.980 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 853 16 1935 1919 0.140588 0 - 0.363217 2024-04-16T08:03:07.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 854 16 1935 1919 0.140424 0 - 0.363217 2024-04-16T08:03:08.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 855 16 1935 1919 0.14026 0 - 0.363217 2024-04-16T08:03:09.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 856 16 1935 1919 0.140096 0 - 0.363217 2024-04-16T08:03:10.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 857 16 1935 1919 0.139932 0 - 0.363217 2024-04-16T08:03:11.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 858 16 1935 1919 0.139769 0 - 0.363217 2024-04-16T08:03:12.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 859 16 1935 1919 0.139606 0 - 0.363217 2024-04-16T08:03:13.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:03:13.982724+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:03:13.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:03:13.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 860 16 1935 1919 0.139444 0 - 0.363217 2024-04-16T08:03:14.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 861 16 1935 1919 0.139282 0 - 0.363217 2024-04-16T08:03:15.981 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 862 16 1935 1919 0.139121 0 - 0.363217 2024-04-16T08:03:16.982 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 863 16 1935 1919 0.138959 0 - 0.363217 2024-04-16T08:03:17.982 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 864 16 1935 1919 0.138799 0 - 0.363217 2024-04-16T08:03:18.982 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 865 16 1935 1919 0.138638 0 - 0.363217 2024-04-16T08:03:19.982 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 866 16 1935 1919 0.138478 0 - 0.363217 2024-04-16T08:03:20.982 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 867 16 1935 1919 0.138318 0 - 0.363217 2024-04-16T08:03:21.982 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 868 16 1935 1919 0.138159 0 - 0.363217 2024-04-16T08:03:22.982 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 869 16 1935 1919 0.138 0 - 0.363217 2024-04-16T08:03:23.982 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 870 16 1935 1919 0.137841 0 - 0.363217 2024-04-16T08:03:24.982 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 871 16 1935 1919 0.137683 0 - 0.363217 2024-04-16T08:03:25.983 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 872 16 1935 1919 0.137525 0 - 0.363217 2024-04-16T08:03:26.983 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 873 16 1935 1919 0.137368 0 - 0.363217 2024-04-16T08:03:27.983 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 874 16 1935 1919 0.13721 0 - 0.363217 2024-04-16T08:03:28.983 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 875 16 1935 1919 0.137054 0 - 0.363217 2024-04-16T08:03:29.983 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 876 16 1935 1919 0.136897 0 - 0.363217 2024-04-16T08:03:30.983 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 877 16 1935 1919 0.136741 0 - 0.363217 2024-04-16T08:03:31.983 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 878 16 1935 1919 0.136585 0 - 0.363217 2024-04-16T08:03:32.983 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 879 16 1935 1919 0.13643 0 - 0.363217 2024-04-16T08:03:33.983 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:03:33.985267+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:03:33.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:03:33.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 880 16 1935 1919 0.136275 0 - 0.363217 2024-04-16T08:03:34.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 881 16 1935 1919 0.13612 0 - 0.363217 2024-04-16T08:03:35.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 882 16 1935 1919 0.135966 0 - 0.363217 2024-04-16T08:03:36.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 883 16 1935 1919 0.135812 0 - 0.363217 2024-04-16T08:03:37.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 884 16 1935 1919 0.135658 0 - 0.363217 2024-04-16T08:03:38.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 885 16 1935 1919 0.135505 0 - 0.363217 2024-04-16T08:03:39.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 886 16 1935 1919 0.135352 0 - 0.363217 2024-04-16T08:03:40.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 887 16 1935 1919 0.135199 0 - 0.363217 2024-04-16T08:03:41.984 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 888 16 1935 1919 0.135047 0 - 0.363217 2024-04-16T08:03:42.985 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 889 16 1935 1919 0.134895 0 - 0.363217 2024-04-16T08:03:43.985 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 890 16 1935 1919 0.134744 0 - 0.363217 2024-04-16T08:03:44.985 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 891 16 1935 1919 0.134593 0 - 0.363217 2024-04-16T08:03:45.985 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 892 16 1935 1919 0.134442 0 - 0.363217 2024-04-16T08:03:46.985 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 893 16 1935 1919 0.134291 0 - 0.363217 2024-04-16T08:03:47.985 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 894 16 1935 1919 0.134141 0 - 0.363217 2024-04-16T08:03:48.985 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 895 16 1935 1919 0.133991 0 - 0.363217 2024-04-16T08:03:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:03:49.035 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:03:49.986 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 896 16 1935 1919 0.133841 0 - 0.363217 2024-04-16T08:03:50.986 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 897 16 1935 1919 0.133692 0 - 0.363217 2024-04-16T08:03:51.037 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T08:03:51.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:03:51.158+0000 7f860535d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:03:51.986 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 898 16 1935 1919 0.133543 0 - 0.363217 2024-04-16T08:03:52.986 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 899 16 1935 1919 0.133395 0 - 0.363217 2024-04-16T08:03:53.986 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:03:53.987910+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:03:53.986 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:03:53.987 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 900 16 1935 1919 0.133247 0 - 0.363217 2024-04-16T08:03:54.986 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 901 16 1935 1919 0.133099 0 - 0.363217 2024-04-16T08:03:55.987 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 902 16 1935 1919 0.132951 0 - 0.363217 2024-04-16T08:03:56.987 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 903 16 1935 1919 0.132804 0 - 0.363217 2024-04-16T08:03:57.987 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 904 16 1935 1919 0.132657 0 - 0.363217 2024-04-16T08:03:58.987 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 905 16 1935 1919 0.13251 0 - 0.363217 2024-04-16T08:03:59.987 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 906 16 1935 1919 0.132364 0 - 0.363217 2024-04-16T08:04:00.987 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 907 16 1935 1919 0.132218 0 - 0.363217 2024-04-16T08:04:01.988 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 908 16 1935 1919 0.132073 0 - 0.363217 2024-04-16T08:04:02.987 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 909 16 1935 1919 0.131927 0 - 0.363217 2024-04-16T08:04:03.988 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 910 16 1935 1919 0.131782 0 - 0.363217 2024-04-16T08:04:04.988 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 911 16 1935 1919 0.131638 0 - 0.363217 2024-04-16T08:04:05.988 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 912 16 1935 1919 0.131493 0 - 0.363217 2024-04-16T08:04:06.988 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 913 16 1935 1919 0.131349 0 - 0.363217 2024-04-16T08:04:07.988 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 914 16 1935 1919 0.131206 0 - 0.363217 2024-04-16T08:04:08.988 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 915 16 1935 1919 0.131062 0 - 0.363217 2024-04-16T08:04:09.988 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 916 16 1935 1919 0.130919 0 - 0.363217 2024-04-16T08:04:10.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 917 16 1935 1919 0.130776 0 - 0.363217 2024-04-16T08:04:11.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 918 16 1935 1919 0.130634 0 - 0.363217 2024-04-16T08:04:12.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 919 16 1935 1919 0.130492 0 - 0.363217 2024-04-16T08:04:13.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:04:13.990563+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:04:13.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:04:13.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 920 16 1935 1919 0.13035 0 - 0.363217 2024-04-16T08:04:14.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 921 16 1935 1919 0.130208 0 - 0.363217 2024-04-16T08:04:15.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 922 16 1935 1919 0.130067 0 - 0.363217 2024-04-16T08:04:16.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 923 16 1935 1919 0.129926 0 - 0.363217 2024-04-16T08:04:17.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 924 16 1935 1919 0.129786 0 - 0.363217 2024-04-16T08:04:18.989 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 925 16 1935 1919 0.129645 0 - 0.363217 2024-04-16T08:04:19.990 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 926 16 1935 1919 0.129505 0 - 0.363217 2024-04-16T08:04:20.990 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 927 16 1935 1919 0.129366 0 - 0.363217 2024-04-16T08:04:21.990 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 928 16 1935 1919 0.129226 0 - 0.363217 2024-04-16T08:04:22.990 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 929 16 1935 1919 0.129087 0 - 0.363217 2024-04-16T08:04:23.990 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 930 16 1935 1919 0.128948 0 - 0.363217 2024-04-16T08:04:24.990 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 931 16 1935 1919 0.12881 0 - 0.363217 2024-04-16T08:04:25.990 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 932 16 1935 1919 0.128672 0 - 0.363217 2024-04-16T08:04:26.991 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 933 16 1935 1919 0.128534 0 - 0.363217 2024-04-16T08:04:27.991 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 934 16 1935 1919 0.128396 0 - 0.363217 2024-04-16T08:04:28.991 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 935 16 1935 1919 0.128259 0 - 0.363217 2024-04-16T08:04:29.991 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 936 16 1935 1919 0.128122 0 - 0.363217 2024-04-16T08:04:30.991 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 937 16 1935 1919 0.127985 0 - 0.363217 2024-04-16T08:04:31.991 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 938 16 1935 1919 0.127849 0 - 0.363217 2024-04-16T08:04:32.991 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 939 16 1935 1919 0.127712 0 - 0.363217 2024-04-16T08:04:33.991 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:04:33.993161+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:04:33.992 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:04:33.992 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 940 16 1935 1919 0.127577 0 - 0.363217 2024-04-16T08:04:34.992 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 941 16 1935 1919 0.127441 0 - 0.363217 2024-04-16T08:04:35.992 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 942 16 1935 1919 0.127306 0 - 0.363217 2024-04-16T08:04:36.992 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 943 16 1935 1919 0.127171 0 - 0.363217 2024-04-16T08:04:37.992 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 944 16 1935 1919 0.127036 0 - 0.363217 2024-04-16T08:04:38.992 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 945 16 1935 1919 0.126902 0 - 0.363217 2024-04-16T08:04:39.992 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 946 16 1935 1919 0.126767 0 - 0.363217 2024-04-16T08:04:40.992 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 947 16 1935 1919 0.126634 0 - 0.363217 2024-04-16T08:04:41.993 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 948 16 1935 1919 0.1265 0 - 0.363217 2024-04-16T08:04:42.993 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 949 16 1935 1919 0.126367 0 - 0.363217 2024-04-16T08:04:43.993 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 950 16 1935 1919 0.126234 0 - 0.363217 2024-04-16T08:04:44.993 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 951 16 1935 1919 0.126101 0 - 0.363217 2024-04-16T08:04:45.993 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 952 16 1935 1919 0.125968 0 - 0.363217 2024-04-16T08:04:46.993 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 953 16 1935 1919 0.125836 0 - 0.363217 2024-04-16T08:04:47.993 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 954 16 1935 1919 0.125704 0 - 0.363217 2024-04-16T08:04:48.993 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 955 16 1935 1919 0.125573 0 - 0.363217 2024-04-16T08:04:49.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 956 16 1935 1919 0.125441 0 - 0.363217 2024-04-16T08:04:50.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 957 16 1935 1919 0.12531 0 - 0.363217 2024-04-16T08:04:51.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 958 16 1935 1919 0.125179 0 - 0.363217 2024-04-16T08:04:52.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 959 16 1935 1919 0.125049 0 - 0.363217 2024-04-16T08:04:53.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:04:53.995737+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:04:53.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:04:53.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 960 16 1935 1919 0.124919 0 - 0.363217 2024-04-16T08:04:54.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 961 16 1935 1919 0.124789 0 - 0.363217 2024-04-16T08:04:55.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 962 16 1935 1919 0.124659 0 - 0.363217 2024-04-16T08:04:56.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 963 16 1935 1919 0.12453 0 - 0.363217 2024-04-16T08:04:57.994 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 964 16 1935 1919 0.1244 0 - 0.363217 2024-04-16T08:04:58.995 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 965 16 1935 1919 0.124271 0 - 0.363217 2024-04-16T08:04:59.995 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 966 16 1935 1919 0.124143 0 - 0.363217 2024-04-16T08:05:00.995 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 967 16 1935 1919 0.124014 0 - 0.363217 2024-04-16T08:05:01.995 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 968 16 1935 1919 0.123886 0 - 0.363217 2024-04-16T08:05:02.995 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 969 16 1935 1919 0.123758 0 - 0.363217 2024-04-16T08:05:03.995 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 970 16 1935 1919 0.123631 0 - 0.363217 2024-04-16T08:05:04.996 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 971 16 1935 1919 0.123504 0 - 0.363217 2024-04-16T08:05:05.996 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 972 16 1935 1919 0.123376 0 - 0.363217 2024-04-16T08:05:06.996 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 973 16 1935 1919 0.12325 0 - 0.363217 2024-04-16T08:05:07.996 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 974 16 1935 1919 0.123123 0 - 0.363217 2024-04-16T08:05:08.996 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 975 16 1935 1919 0.122997 0 - 0.363217 2024-04-16T08:05:09.996 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 976 16 1935 1919 0.122871 0 - 0.363217 2024-04-16T08:05:10.996 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 977 16 1935 1919 0.122745 0 - 0.363217 2024-04-16T08:05:11.996 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 978 16 1935 1919 0.12262 0 - 0.363217 2024-04-16T08:05:12.997 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 979 16 1935 1919 0.122494 0 - 0.363217 2024-04-16T08:05:13.997 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:05:13.998420+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:05:13.997 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:05:13.997 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 980 16 1935 1919 0.122369 0 - 0.363217 2024-04-16T08:05:14.997 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 981 16 1935 1919 0.122245 0 - 0.363217 2024-04-16T08:05:15.997 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 982 16 1935 1919 0.12212 0 - 0.363217 2024-04-16T08:05:16.997 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 983 16 1935 1919 0.121996 0 - 0.363217 2024-04-16T08:05:17.998 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 984 16 1935 1919 0.121872 0 - 0.363217 2024-04-16T08:05:18.997 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 985 16 1935 1919 0.121748 0 - 0.363217 2024-04-16T08:05:19.998 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 986 16 1935 1919 0.121625 0 - 0.363217 2024-04-16T08:05:20.998 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 987 16 1935 1919 0.121501 0 - 0.363217 2024-04-16T08:05:21.998 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 988 16 1935 1919 0.121378 0 - 0.363217 2024-04-16T08:05:22.998 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 989 16 1935 1919 0.121256 0 - 0.363217 2024-04-16T08:05:23.998 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 990 16 1935 1919 0.121133 0 - 0.363217 2024-04-16T08:05:24.998 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 991 16 1935 1919 0.121011 0 - 0.363217 2024-04-16T08:05:25.999 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 992 16 1935 1919 0.120889 0 - 0.363217 2024-04-16T08:05:26.998 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 993 16 1935 1919 0.120767 0 - 0.363217 2024-04-16T08:05:27.998 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 994 16 1935 1919 0.120646 0 - 0.363217 2024-04-16T08:05:28.999 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 995 16 1935 1919 0.120525 0 - 0.363217 2024-04-16T08:05:29.999 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 996 16 1935 1919 0.120404 0 - 0.363217 2024-04-16T08:05:30.999 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 997 16 1935 1919 0.120283 0 - 0.363217 2024-04-16T08:05:31.999 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 998 16 1935 1919 0.120162 0 - 0.363217 2024-04-16T08:05:32.999 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 999 16 1935 1919 0.120042 0 - 0.363217 2024-04-16T08:05:33.999 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:05:34.001126+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:05:34.000 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:05:34.000 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1000 16 1935 1919 0.119922 0 - 0.363217 2024-04-16T08:05:34.999 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1001 16 1935 1919 0.119802 0 - 0.363217 2024-04-16T08:05:36.000 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1002 16 1935 1919 0.119683 0 - 0.363217 2024-04-16T08:05:37.000 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1003 16 1935 1919 0.119563 0 - 0.363217 2024-04-16T08:05:38.000 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1004 16 1935 1919 0.119444 0 - 0.363217 2024-04-16T08:05:39.000 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1005 16 1935 1919 0.119325 0 - 0.363217 2024-04-16T08:05:40.001 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1006 16 1935 1919 0.119207 0 - 0.363217 2024-04-16T08:05:41.000 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1007 16 1935 1919 0.119088 0 - 0.363217 2024-04-16T08:05:42.000 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1008 16 1935 1919 0.11897 0 - 0.363217 2024-04-16T08:05:43.001 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1009 16 1935 1919 0.118852 0 - 0.363217 2024-04-16T08:05:44.001 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1010 16 1935 1919 0.118735 0 - 0.363217 2024-04-16T08:05:45.001 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1011 16 1935 1919 0.118617 0 - 0.363217 2024-04-16T08:05:46.001 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1012 16 1935 1919 0.1185 0 - 0.363217 2024-04-16T08:05:47.001 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1013 16 1935 1919 0.118383 0 - 0.363217 2024-04-16T08:05:48.001 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1014 16 1935 1919 0.118266 0 - 0.363217 2024-04-16T08:05:49.001 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1015 16 1935 1919 0.11815 0 - 0.363217 2024-04-16T08:05:50.002 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1016 16 1935 1919 0.118033 0 - 0.363217 2024-04-16T08:05:51.002 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1017 16 1935 1919 0.117917 0 - 0.363217 2024-04-16T08:05:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:05:51.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:05:52.002 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1018 16 1935 1919 0.117801 0 - 0.363217 2024-04-16T08:05:53.002 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1019 16 1935 1919 0.117686 0 - 0.363217 2024-04-16T08:05:53.078 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:05:53.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:05:53.200+0000 7fec35f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:05:54.002 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:05:54.003831+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:05:54.002 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:05:54.002 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1020 16 1935 1919 0.11757 0 - 0.363217 2024-04-16T08:05:55.002 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1021 16 1935 1919 0.117455 0 - 0.363217 2024-04-16T08:05:56.002 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1022 16 1935 1919 0.11734 0 - 0.363217 2024-04-16T08:05:57.003 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1023 16 1935 1919 0.117226 0 - 0.363217 2024-04-16T08:05:58.003 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1024 16 1935 1919 0.117111 0 - 0.363217 2024-04-16T08:05:59.003 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1025 16 1935 1919 0.116997 0 - 0.363217 2024-04-16T08:06:00.006 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1026 16 1935 1919 0.116883 0 - 0.363217 2024-04-16T08:06:01.003 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1027 16 1935 1919 0.116769 0 - 0.363217 2024-04-16T08:06:02.003 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1028 16 1935 1919 0.116656 0 - 0.363217 2024-04-16T08:06:03.003 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1029 16 1935 1919 0.116542 0 - 0.363217 2024-04-16T08:06:04.003 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1030 16 1935 1919 0.116429 0 - 0.363217 2024-04-16T08:06:05.004 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1031 16 1935 1919 0.116316 0 - 0.363217 2024-04-16T08:06:06.004 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1032 16 1935 1919 0.116203 0 - 0.363217 2024-04-16T08:06:07.004 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1033 16 1935 1919 0.116091 0 - 0.363217 2024-04-16T08:06:08.004 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1034 16 1935 1919 0.115979 0 - 0.363217 2024-04-16T08:06:09.004 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1035 16 1935 1919 0.115867 0 - 0.363217 2024-04-16T08:06:10.004 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1036 16 1935 1919 0.115755 0 - 0.363217 2024-04-16T08:06:11.004 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1037 16 1935 1919 0.115643 0 - 0.363217 2024-04-16T08:06:12.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1038 16 1935 1919 0.115532 0 - 0.363217 2024-04-16T08:06:13.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1039 16 1935 1919 0.11542 0 - 0.363217 2024-04-16T08:06:14.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:06:14.006558+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:06:14.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:06:14.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1040 16 1935 1919 0.11531 0 - 0.363217 2024-04-16T08:06:15.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1041 16 1935 1919 0.115199 0 - 0.363217 2024-04-16T08:06:16.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1042 16 1935 1919 0.115088 0 - 0.363217 2024-04-16T08:06:17.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1043 16 1935 1919 0.114978 0 - 0.363217 2024-04-16T08:06:18.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1044 16 1935 1919 0.114868 0 - 0.363217 2024-04-16T08:06:19.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1045 16 1935 1919 0.114758 0 - 0.363217 2024-04-16T08:06:20.005 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1046 16 1935 1919 0.114648 0 - 0.363217 2024-04-16T08:06:21.006 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1047 16 1935 1919 0.114539 0 - 0.363217 2024-04-16T08:06:22.006 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1048 16 1935 1919 0.114429 0 - 0.363217 2024-04-16T08:06:23.006 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1049 16 1935 1919 0.11432 0 - 0.363217 2024-04-16T08:06:24.006 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1050 16 1935 1919 0.114211 0 - 0.363217 2024-04-16T08:06:25.006 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1051 16 1935 1919 0.114103 0 - 0.363217 2024-04-16T08:06:26.006 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1052 16 1935 1919 0.113994 0 - 0.363217 2024-04-16T08:06:27.007 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1053 16 1935 1919 0.113886 0 - 0.363217 2024-04-16T08:06:28.007 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1054 16 1935 1919 0.113778 0 - 0.363217 2024-04-16T08:06:29.007 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1055 16 1935 1919 0.11367 0 - 0.363217 2024-04-16T08:06:30.007 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1056 16 1935 1919 0.113562 0 - 0.363217 2024-04-16T08:06:31.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1057 16 1935 1919 0.113455 0 - 0.363217 2024-04-16T08:06:32.007 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1058 16 1935 1919 0.113348 0 - 0.363217 2024-04-16T08:06:33.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1059 16 1935 1919 0.113241 0 - 0.363217 2024-04-16T08:06:34.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:06:34.009180+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:06:34.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:06:34.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1060 16 1935 1919 0.113134 0 - 0.363217 2024-04-16T08:06:35.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1061 16 1935 1919 0.113027 0 - 0.363217 2024-04-16T08:06:36.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1062 16 1935 1919 0.112921 0 - 0.363217 2024-04-16T08:06:37.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1063 16 1935 1919 0.112815 0 - 0.363217 2024-04-16T08:06:38.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1064 16 1935 1919 0.112709 0 - 0.363217 2024-04-16T08:06:39.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1065 16 1935 1919 0.112603 0 - 0.363217 2024-04-16T08:06:40.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1066 16 1935 1919 0.112497 0 - 0.363217 2024-04-16T08:06:41.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1067 16 1935 1919 0.112392 0 - 0.363217 2024-04-16T08:06:42.008 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1068 16 1935 1919 0.112286 0 - 0.363217 2024-04-16T08:06:43.009 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1069 16 1935 1919 0.112181 0 - 0.363217 2024-04-16T08:06:44.009 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1070 16 1935 1919 0.112077 0 - 0.363217 2024-04-16T08:06:45.009 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1071 16 1935 1919 0.111972 0 - 0.363217 2024-04-16T08:06:46.009 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1072 16 1935 1919 0.111867 0 - 0.363217 2024-04-16T08:06:47.009 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1073 16 1935 1919 0.111763 0 - 0.363217 2024-04-16T08:06:48.009 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1074 16 1935 1919 0.111659 0 - 0.363217 2024-04-16T08:06:49.009 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1075 16 1935 1919 0.111555 0 - 0.363217 2024-04-16T08:06:50.009 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1076 16 1935 1919 0.111452 0 - 0.363217 2024-04-16T08:06:51.010 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1077 16 1935 1919 0.111348 0 - 0.363217 2024-04-16T08:06:52.010 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1078 16 1935 1919 0.111245 0 - 0.363217 2024-04-16T08:06:53.010 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1079 16 1935 1919 0.111142 0 - 0.363217 2024-04-16T08:06:54.010 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:06:54.011842+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:06:54.010 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:06:54.010 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1080 16 1935 1919 0.111039 0 - 0.363217 2024-04-16T08:06:55.010 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1081 16 1935 1919 0.110936 0 - 0.363217 2024-04-16T08:06:56.010 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1082 16 1935 1919 0.110834 0 - 0.363217 2024-04-16T08:06:57.010 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1083 16 1935 1919 0.110731 0 - 0.363217 2024-04-16T08:06:58.011 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1084 16 1935 1919 0.110629 0 - 0.363217 2024-04-16T08:06:59.011 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1085 16 1935 1919 0.110527 0 - 0.363217 2024-04-16T08:07:00.011 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1086 16 1935 1919 0.110425 0 - 0.363217 2024-04-16T08:07:01.011 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1087 16 1935 1919 0.110324 0 - 0.363217 2024-04-16T08:07:02.011 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1088 16 1935 1919 0.110222 0 - 0.363217 2024-04-16T08:07:03.011 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1089 16 1935 1919 0.110121 0 - 0.363217 2024-04-16T08:07:04.011 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1090 16 1935 1919 0.11002 0 - 0.363217 2024-04-16T08:07:05.012 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1091 16 1935 1919 0.109919 0 - 0.363217 2024-04-16T08:07:06.012 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1092 16 1935 1919 0.109819 0 - 0.363217 2024-04-16T08:07:07.012 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1093 16 1935 1919 0.109718 0 - 0.363217 2024-04-16T08:07:08.012 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1094 16 1935 1919 0.109618 0 - 0.363217 2024-04-16T08:07:09.012 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1095 16 1935 1919 0.109518 0 - 0.363217 2024-04-16T08:07:10.012 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1096 16 1935 1919 0.109418 0 - 0.363217 2024-04-16T08:07:11.012 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1097 16 1935 1919 0.109318 0 - 0.363217 2024-04-16T08:07:12.012 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1098 16 1935 1919 0.109218 0 - 0.363217 2024-04-16T08:07:13.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1099 16 1935 1919 0.109119 0 - 0.363217 2024-04-16T08:07:14.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:07:14.014454+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:07:14.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:07:14.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1100 16 1935 1919 0.10902 0 - 0.363217 2024-04-16T08:07:15.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1101 16 1935 1919 0.108921 0 - 0.363217 2024-04-16T08:07:16.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1102 16 1935 1919 0.108822 0 - 0.363217 2024-04-16T08:07:17.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1103 16 1935 1919 0.108723 0 - 0.363217 2024-04-16T08:07:18.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1104 16 1935 1919 0.108625 0 - 0.363217 2024-04-16T08:07:19.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1105 16 1935 1919 0.108527 0 - 0.363217 2024-04-16T08:07:20.013 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1106 16 1935 1919 0.108428 0 - 0.363217 2024-04-16T08:07:21.014 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1107 16 1935 1919 0.108331 0 - 0.363217 2024-04-16T08:07:22.014 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1108 16 1935 1919 0.108233 0 - 0.363217 2024-04-16T08:07:23.014 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1109 16 1935 1919 0.108135 0 - 0.363217 2024-04-16T08:07:24.014 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1110 16 1935 1919 0.108038 0 - 0.363217 2024-04-16T08:07:25.014 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1111 16 1935 1919 0.10794 0 - 0.363217 2024-04-16T08:07:26.014 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1112 16 1935 1919 0.107843 0 - 0.363217 2024-04-16T08:07:27.014 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1113 16 1935 1919 0.107747 0 - 0.363217 2024-04-16T08:07:28.015 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1114 16 1935 1919 0.10765 0 - 0.363217 2024-04-16T08:07:29.015 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1115 16 1935 1919 0.107553 0 - 0.363217 2024-04-16T08:07:30.015 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1116 16 1935 1919 0.107457 0 - 0.363217 2024-04-16T08:07:31.015 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1117 16 1935 1919 0.107361 0 - 0.363217 2024-04-16T08:07:32.015 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1118 16 1935 1919 0.107265 0 - 0.363217 2024-04-16T08:07:33.015 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1119 16 1935 1919 0.107169 0 - 0.363217 2024-04-16T08:07:34.015 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:07:34.017123+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:07:34.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:07:34.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1120 16 1935 1919 0.107073 0 - 0.363217 2024-04-16T08:07:35.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1121 16 1935 1919 0.106978 0 - 0.363217 2024-04-16T08:07:36.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1122 16 1935 1919 0.106882 0 - 0.363217 2024-04-16T08:07:37.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1123 16 1935 1919 0.106787 0 - 0.363217 2024-04-16T08:07:38.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1124 16 1935 1919 0.106692 0 - 0.363217 2024-04-16T08:07:39.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1125 16 1935 1919 0.106597 0 - 0.363217 2024-04-16T08:07:40.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1126 16 1935 1919 0.106503 0 - 0.363217 2024-04-16T08:07:41.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1127 16 1935 1919 0.106408 0 - 0.363217 2024-04-16T08:07:42.016 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1128 16 1935 1919 0.106314 0 - 0.363217 2024-04-16T08:07:43.017 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1129 16 1935 1919 0.10622 0 - 0.363217 2024-04-16T08:07:44.017 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1130 16 1935 1919 0.106126 0 - 0.363217 2024-04-16T08:07:45.017 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1131 16 1935 1919 0.106032 0 - 0.363217 2024-04-16T08:07:46.017 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1132 16 1935 1919 0.105938 0 - 0.363217 2024-04-16T08:07:47.017 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1133 16 1935 1919 0.105845 0 - 0.363217 2024-04-16T08:07:48.017 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1134 16 1935 1919 0.105751 0 - 0.363217 2024-04-16T08:07:49.017 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1135 16 1935 1919 0.105658 0 - 0.363217 2024-04-16T08:07:50.018 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1136 16 1935 1919 0.105565 0 - 0.363217 2024-04-16T08:07:51.018 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1137 16 1935 1919 0.105472 0 - 0.363217 2024-04-16T08:07:52.018 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1138 16 1935 1919 0.105379 0 - 0.363217 2024-04-16T08:07:53.018 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1139 16 1935 1919 0.105287 0 - 0.363217 2024-04-16T08:07:53.117 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:07:53.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:07:54.018 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:07:54.019861+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:07:54.018 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:07:54.018 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1140 16 1935 1919 0.105195 0 - 0.363217 2024-04-16T08:07:55.018 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1141 16 1935 1919 0.105102 0 - 0.363217 2024-04-16T08:07:55.119 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T08:07:55.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:07:55.241+0000 7f4b00e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:07:56.018 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1142 16 1935 1919 0.10501 0 - 0.363217 2024-04-16T08:07:57.019 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1143 16 1935 1919 0.104919 0 - 0.363217 2024-04-16T08:07:58.019 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1144 16 1935 1919 0.104827 0 - 0.363217 2024-04-16T08:07:59.019 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1145 16 1935 1919 0.104735 0 - 0.363217 2024-04-16T08:08:00.019 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1146 16 1935 1919 0.104644 0 - 0.363217 2024-04-16T08:08:01.019 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1147 16 1935 1919 0.104553 0 - 0.363217 2024-04-16T08:08:02.019 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1148 16 1935 1919 0.104462 0 - 0.363217 2024-04-16T08:08:03.019 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1149 16 1935 1919 0.104371 0 - 0.363217 2024-04-16T08:08:04.019 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1150 16 1935 1919 0.10428 0 - 0.363217 2024-04-16T08:08:05.020 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1151 16 1935 1919 0.104189 0 - 0.363217 2024-04-16T08:08:06.020 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1152 16 1935 1919 0.104099 0 - 0.363217 2024-04-16T08:08:07.020 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1153 16 1935 1919 0.104009 0 - 0.363217 2024-04-16T08:08:08.020 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1154 16 1935 1919 0.103918 0 - 0.363217 2024-04-16T08:08:09.020 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1155 16 1935 1919 0.103828 0 - 0.363217 2024-04-16T08:08:10.020 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1156 16 1935 1919 0.103739 0 - 0.363217 2024-04-16T08:08:11.020 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1157 16 1935 1919 0.103649 0 - 0.363217 2024-04-16T08:08:12.020 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1158 16 1935 1919 0.103559 0 - 0.363217 2024-04-16T08:08:13.020 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1159 16 1935 1919 0.10347 0 - 0.363217 2024-04-16T08:08:14.021 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:08:14.022546+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:08:14.021 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:08:14.021 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1160 16 1935 1919 0.103381 0 - 0.363217 2024-04-16T08:08:15.021 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1161 16 1935 1919 0.103292 0 - 0.363217 2024-04-16T08:08:16.021 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1162 16 1935 1919 0.103203 0 - 0.363217 2024-04-16T08:08:17.021 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1163 16 1935 1919 0.103114 0 - 0.363217 2024-04-16T08:08:18.021 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1164 16 1935 1919 0.103026 0 - 0.363217 2024-04-16T08:08:19.021 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1165 16 1935 1919 0.102937 0 - 0.363217 2024-04-16T08:08:20.022 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1166 16 1935 1919 0.102849 0 - 0.363217 2024-04-16T08:08:21.022 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1167 16 1935 1919 0.102761 0 - 0.363217 2024-04-16T08:08:22.022 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1168 16 1935 1919 0.102673 0 - 0.363217 2024-04-16T08:08:23.022 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1169 16 1935 1919 0.102585 0 - 0.363217 2024-04-16T08:08:24.022 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1170 16 1935 1919 0.102497 0 - 0.363217 2024-04-16T08:08:25.022 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1171 16 1935 1919 0.10241 0 - 0.363217 2024-04-16T08:08:26.022 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1172 16 1935 1919 0.102322 0 - 0.363217 2024-04-16T08:08:27.022 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1173 16 1935 1919 0.102235 0 - 0.363217 2024-04-16T08:08:28.023 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1174 16 1935 1919 0.102148 0 - 0.363217 2024-04-16T08:08:29.023 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1175 16 1935 1919 0.102061 0 - 0.363217 2024-04-16T08:08:30.023 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1176 16 1935 1919 0.101974 0 - 0.363217 2024-04-16T08:08:31.023 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1177 16 1935 1919 0.101888 0 - 0.363217 2024-04-16T08:08:32.023 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1178 16 1935 1919 0.101801 0 - 0.363217 2024-04-16T08:08:33.023 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1179 16 1935 1919 0.101715 0 - 0.363217 2024-04-16T08:08:34.023 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:08:34.025251+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:08:34.024 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:08:34.024 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1180 16 1935 1919 0.101629 0 - 0.363217 2024-04-16T08:08:35.023 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1181 16 1935 1919 0.101543 0 - 0.363217 2024-04-16T08:08:36.024 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1182 16 1935 1919 0.101457 0 - 0.363217 2024-04-16T08:08:37.024 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1183 16 1935 1919 0.101371 0 - 0.363217 2024-04-16T08:08:38.024 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1184 16 1935 1919 0.101285 0 - 0.363217 2024-04-16T08:08:39.024 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1185 16 1935 1919 0.1012 0 - 0.363217 2024-04-16T08:08:40.024 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1186 16 1935 1919 0.101115 0 - 0.363217 2024-04-16T08:08:41.024 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1187 16 1935 1919 0.101029 0 - 0.363217 2024-04-16T08:08:42.024 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1188 16 1935 1919 0.100944 0 - 0.363217 2024-04-16T08:08:43.025 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1189 16 1935 1919 0.100859 0 - 0.363217 2024-04-16T08:08:44.025 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1190 16 1935 1919 0.100775 0 - 0.363217 2024-04-16T08:08:45.025 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1191 16 1935 1919 0.10069 0 - 0.363217 2024-04-16T08:08:46.025 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1192 16 1935 1919 0.100606 0 - 0.363217 2024-04-16T08:08:47.025 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1193 16 1935 1919 0.100521 0 - 0.363217 2024-04-16T08:08:48.025 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1194 16 1935 1919 0.100437 0 - 0.363217 2024-04-16T08:08:49.025 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1195 16 1935 1919 0.100353 0 - 0.363217 2024-04-16T08:08:50.026 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1196 16 1935 1919 0.100269 0 - 0.363217 2024-04-16T08:08:51.026 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1197 16 1935 1919 0.100185 0 - 0.363217 2024-04-16T08:08:52.026 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1198 16 1935 1919 0.100102 0 - 0.363217 2024-04-16T08:08:53.026 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1199 16 1935 1919 0.100018 0 - 0.363217 2024-04-16T08:08:54.026 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:08:54.027828+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:08:54.026 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:08:54.026 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1200 16 1935 1919 0.0999349 0 - 0.363217 2024-04-16T08:08:55.026 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1201 16 1935 1919 0.0998517 0 - 0.363217 2024-04-16T08:08:56.026 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1202 16 1935 1919 0.0997686 0 - 0.363217 2024-04-16T08:08:57.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1203 16 1935 1919 0.0996857 0 - 0.363217 2024-04-16T08:08:58.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1204 16 1935 1919 0.0996029 0 - 0.363217 2024-04-16T08:08:59.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1205 16 1935 1919 0.0995202 0 - 0.363217 2024-04-16T08:09:00.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1206 16 1935 1919 0.0994377 0 - 0.363217 2024-04-16T08:09:01.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1207 16 1935 1919 0.0993553 0 - 0.363217 2024-04-16T08:09:02.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1208 16 1935 1919 0.0992731 0 - 0.363217 2024-04-16T08:09:03.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1209 16 1935 1919 0.0991909 0 - 0.363217 2024-04-16T08:09:04.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1210 16 1935 1919 0.099109 0 - 0.363217 2024-04-16T08:09:05.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1211 16 1935 1919 0.0990271 0 - 0.363217 2024-04-16T08:09:06.027 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1212 16 1935 1919 0.0989454 0 - 0.363217 2024-04-16T08:09:07.028 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1213 16 1935 1919 0.0988639 0 - 0.363217 2024-04-16T08:09:08.028 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1214 16 1935 1919 0.0987824 0 - 0.363217 2024-04-16T08:09:09.028 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1215 16 1935 1919 0.0987011 0 - 0.363217 2024-04-16T08:09:10.028 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1216 16 1935 1919 0.0986199 0 - 0.363217 2024-04-16T08:09:11.028 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1217 16 1935 1919 0.0985389 0 - 0.363217 2024-04-16T08:09:12.028 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1218 16 1935 1919 0.098458 0 - 0.363217 2024-04-16T08:09:13.029 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1219 16 1935 1919 0.0983772 0 - 0.363217 2024-04-16T08:09:14.029 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:09:14.030423+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:09:14.029 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:09:14.029 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1220 16 1935 1919 0.0982966 0 - 0.363217 2024-04-16T08:09:15.029 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1221 16 1935 1919 0.0982161 0 - 0.363217 2024-04-16T08:09:16.029 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1222 16 1935 1919 0.0981357 0 - 0.363217 2024-04-16T08:09:17.029 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1223 16 1935 1919 0.0980555 0 - 0.363217 2024-04-16T08:09:18.030 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1224 16 1935 1919 0.0979754 0 - 0.363217 2024-04-16T08:09:19.029 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1225 16 1935 1919 0.0978954 0 - 0.363217 2024-04-16T08:09:20.029 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1226 16 1935 1919 0.0978155 0 - 0.363217 2024-04-16T08:09:21.030 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1227 16 1935 1919 0.0977358 0 - 0.363217 2024-04-16T08:09:22.030 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1228 16 1935 1919 0.0976562 0 - 0.363217 2024-04-16T08:09:23.030 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1229 16 1935 1919 0.0975768 0 - 0.363217 2024-04-16T08:09:24.030 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1230 16 1935 1919 0.0974974 0 - 0.363217 2024-04-16T08:09:25.030 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1231 16 1935 1919 0.0974182 0 - 0.363217 2024-04-16T08:09:26.030 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1232 16 1935 1919 0.0973392 0 - 0.363217 2024-04-16T08:09:27.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1233 16 1935 1919 0.0972602 0 - 0.363217 2024-04-16T08:09:28.030 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1234 16 1935 1919 0.0971814 0 - 0.363217 2024-04-16T08:09:29.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1235 16 1935 1919 0.0971027 0 - 0.363217 2024-04-16T08:09:30.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1236 16 1935 1919 0.0970241 0 - 0.363217 2024-04-16T08:09:31.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1237 16 1935 1919 0.0969457 0 - 0.363217 2024-04-16T08:09:32.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1238 16 1935 1919 0.0968674 0 - 0.363217 2024-04-16T08:09:33.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1239 16 1935 1919 0.0967892 0 - 0.363217 2024-04-16T08:09:34.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:09:34.033020+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:09:34.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:09:34.032 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1240 16 1935 1919 0.0967112 0 - 0.363217 2024-04-16T08:09:35.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1241 16 1935 1919 0.0966332 0 - 0.363217 2024-04-16T08:09:36.031 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1242 16 1935 1919 0.0965554 0 - 0.363217 2024-04-16T08:09:37.032 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1243 16 1935 1919 0.0964778 0 - 0.363217 2024-04-16T08:09:38.032 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1244 16 1935 1919 0.0964002 0 - 0.363217 2024-04-16T08:09:39.032 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1245 16 1935 1919 0.0963228 0 - 0.363217 2024-04-16T08:09:40.033 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1246 16 1935 1919 0.0962455 0 - 0.363217 2024-04-16T08:09:41.032 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1247 16 1935 1919 0.0961683 0 - 0.363217 2024-04-16T08:09:42.032 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1248 16 1935 1919 0.0960912 0 - 0.363217 2024-04-16T08:09:43.032 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1249 16 1935 1919 0.0960143 0 - 0.363217 2024-04-16T08:09:44.032 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1250 16 1935 1919 0.0959375 0 - 0.363217 2024-04-16T08:09:45.033 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1251 16 1935 1919 0.0958608 0 - 0.363217 2024-04-16T08:09:46.033 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1252 16 1935 1919 0.0957842 0 - 0.363217 2024-04-16T08:09:47.033 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1253 16 1935 1919 0.0957078 0 - 0.363217 2024-04-16T08:09:48.033 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1254 16 1935 1919 0.0956315 0 - 0.363217 2024-04-16T08:09:49.033 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1255 16 1935 1919 0.0955553 0 - 0.363217 2024-04-16T08:09:50.033 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1256 16 1935 1919 0.0954792 0 - 0.363217 2024-04-16T08:09:51.033 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1257 16 1935 1919 0.0954032 0 - 0.363217 2024-04-16T08:09:52.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1258 16 1935 1919 0.0953274 0 - 0.363217 2024-04-16T08:09:53.034 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1259 16 1935 1919 0.0952517 0 - 0.363217 2024-04-16T08:09:54.034 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:09:54.035612+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:09:54.034 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:09:54.034 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1260 16 1935 1919 0.0951761 0 - 0.363217 2024-04-16T08:09:55.034 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1261 16 1935 1919 0.0951006 0 - 0.363217 2024-04-16T08:09:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:09:55.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:09:56.034 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1262 16 1935 1919 0.0950252 0 - 0.363217 2024-04-16T08:09:57.034 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1263 16 1935 1919 0.09495 0 - 0.363217 2024-04-16T08:09:57.160 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:09:57.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:09:57.281+0000 7f55d80ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:09:58.034 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1264 16 1935 1919 0.0948749 0 - 0.363217 2024-04-16T08:09:59.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1265 16 1935 1919 0.0947999 0 - 0.363217 2024-04-16T08:10:00.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1266 16 1935 1919 0.094725 0 - 0.363217 2024-04-16T08:10:01.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1267 16 1935 1919 0.0946502 0 - 0.363217 2024-04-16T08:10:02.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1268 16 1935 1919 0.0945756 0 - 0.363217 2024-04-16T08:10:03.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1269 16 1935 1919 0.0945011 0 - 0.363217 2024-04-16T08:10:04.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1270 16 1935 1919 0.0944267 0 - 0.363217 2024-04-16T08:10:05.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1271 16 1935 1919 0.0943524 0 - 0.363217 2024-04-16T08:10:06.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1272 16 1935 1919 0.0942782 0 - 0.363217 2024-04-16T08:10:07.035 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1273 16 1935 1919 0.0942041 0 - 0.363217 2024-04-16T08:10:08.036 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1274 16 1935 1919 0.0941302 0 - 0.363217 2024-04-16T08:10:09.036 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1275 16 1935 1919 0.0940564 0 - 0.363217 2024-04-16T08:10:10.036 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1276 16 1935 1919 0.0939826 0 - 0.363217 2024-04-16T08:10:11.036 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1277 16 1935 1919 0.093909 0 - 0.363217 2024-04-16T08:10:12.036 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1278 16 1935 1919 0.0938356 0 - 0.363217 2024-04-16T08:10:13.036 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1279 16 1935 1919 0.0937622 0 - 0.363217 2024-04-16T08:10:14.036 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:10:14.038277+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:10:14.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:10:14.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1280 16 1935 1919 0.0936889 0 - 0.363217 2024-04-16T08:10:15.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1281 16 1935 1919 0.0936158 0 - 0.363217 2024-04-16T08:10:16.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1282 16 1935 1919 0.0935428 0 - 0.363217 2024-04-16T08:10:17.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1283 16 1935 1919 0.0934699 0 - 0.363217 2024-04-16T08:10:18.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1284 16 1935 1919 0.0933971 0 - 0.363217 2024-04-16T08:10:19.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1285 16 1935 1919 0.0933244 0 - 0.363217 2024-04-16T08:10:20.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1286 16 1935 1919 0.0932518 0 - 0.363217 2024-04-16T08:10:21.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1287 16 1935 1919 0.0931794 0 - 0.363217 2024-04-16T08:10:22.037 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1288 16 1935 1919 0.093107 0 - 0.363217 2024-04-16T08:10:23.038 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1289 16 1935 1919 0.0930348 0 - 0.363217 2024-04-16T08:10:24.038 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1290 16 1935 1919 0.0929627 0 - 0.363217 2024-04-16T08:10:25.038 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1291 16 1935 1919 0.0928907 0 - 0.363217 2024-04-16T08:10:26.038 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1292 16 1935 1919 0.0928188 0 - 0.363217 2024-04-16T08:10:27.038 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1293 16 1935 1919 0.092747 0 - 0.363217 2024-04-16T08:10:28.038 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1294 16 1935 1919 0.0926753 0 - 0.363217 2024-04-16T08:10:29.038 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1295 16 1935 1919 0.0926037 0 - 0.363217 2024-04-16T08:10:30.038 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1296 16 1935 1919 0.0925323 0 - 0.363217 2024-04-16T08:10:31.038 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1297 16 1935 1919 0.0924609 0 - 0.363217 2024-04-16T08:10:32.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1298 16 1935 1919 0.0923897 0 - 0.363217 2024-04-16T08:10:33.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1299 16 1935 1919 0.0923186 0 - 0.363217 2024-04-16T08:10:34.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:10:34.040849+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:10:34.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:10:34.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1300 16 1935 1919 0.0922476 0 - 0.363217 2024-04-16T08:10:35.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1301 16 1935 1919 0.0921767 0 - 0.363217 2024-04-16T08:10:36.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1302 16 1935 1919 0.0921059 0 - 0.363217 2024-04-16T08:10:37.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1303 16 1935 1919 0.0920352 0 - 0.363217 2024-04-16T08:10:38.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1304 16 1935 1919 0.0919646 0 - 0.363217 2024-04-16T08:10:39.039 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1305 16 1935 1919 0.0918941 0 - 0.363217 2024-04-16T08:10:40.040 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1306 16 1935 1919 0.0918238 0 - 0.363217 2024-04-16T08:10:41.040 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1307 16 1935 1919 0.0917535 0 - 0.363217 2024-04-16T08:10:42.040 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1308 16 1935 1919 0.0916834 0 - 0.363217 2024-04-16T08:10:43.040 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1309 16 1935 1919 0.0916133 0 - 0.363217 2024-04-16T08:10:44.040 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1310 16 1935 1919 0.0915434 0 - 0.363217 2024-04-16T08:10:45.040 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1311 16 1935 1919 0.0914736 0 - 0.363217 2024-04-16T08:10:46.040 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1312 16 1935 1919 0.0914039 0 - 0.363217 2024-04-16T08:10:47.040 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1313 16 1935 1919 0.0913342 0 - 0.363217 2024-04-16T08:10:48.041 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1314 16 1935 1919 0.0912647 0 - 0.363217 2024-04-16T08:10:49.041 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1315 16 1935 1919 0.0911953 0 - 0.363217 2024-04-16T08:10:50.041 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1316 16 1935 1919 0.091126 0 - 0.363217 2024-04-16T08:10:51.041 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1317 16 1935 1919 0.0910568 0 - 0.363217 2024-04-16T08:10:52.041 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1318 16 1935 1919 0.0909878 0 - 0.363217 2024-04-16T08:10:53.041 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1319 16 1935 1919 0.0909188 0 - 0.363217 2024-04-16T08:10:54.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:10:54.043410+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:10:54.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:10:54.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1320 16 1935 1919 0.0908499 0 - 0.363217 2024-04-16T08:10:55.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1321 16 1935 1919 0.0907811 0 - 0.363217 2024-04-16T08:10:56.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1322 16 1935 1919 0.0907124 0 - 0.363217 2024-04-16T08:10:57.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1323 16 1935 1919 0.0906439 0 - 0.363217 2024-04-16T08:10:58.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1324 16 1935 1919 0.0905754 0 - 0.363217 2024-04-16T08:10:59.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1325 16 1935 1919 0.0905071 0 - 0.363217 2024-04-16T08:11:00.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1326 16 1935 1919 0.0904388 0 - 0.363217 2024-04-16T08:11:01.042 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1327 16 1935 1919 0.0903706 0 - 0.363217 2024-04-16T08:11:02.043 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1328 16 1935 1919 0.0903026 0 - 0.363217 2024-04-16T08:11:03.043 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1329 16 1935 1919 0.0902346 0 - 0.363217 2024-04-16T08:11:04.043 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1330 16 1935 1919 0.0901668 0 - 0.363217 2024-04-16T08:11:05.043 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1331 16 1935 1919 0.0900991 0 - 0.363217 2024-04-16T08:11:06.043 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1332 16 1935 1919 0.0900314 0 - 0.363217 2024-04-16T08:11:07.043 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1333 16 1935 1919 0.0899639 0 - 0.363217 2024-04-16T08:11:08.043 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1334 16 1935 1919 0.0898964 0 - 0.363217 2024-04-16T08:11:09.044 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1335 16 1935 1919 0.0898291 0 - 0.363217 2024-04-16T08:11:10.044 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1336 16 1935 1919 0.0897619 0 - 0.363217 2024-04-16T08:11:11.044 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1337 16 1935 1919 0.0896947 0 - 0.363217 2024-04-16T08:11:12.044 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1338 16 1935 1919 0.0896277 0 - 0.363217 2024-04-16T08:11:13.044 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1339 16 1935 1919 0.0895608 0 - 0.363217 2024-04-16T08:11:14.044 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:11:14.046133+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:11:14.045 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:11:14.045 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1340 16 1935 1919 0.0894939 0 - 0.363217 2024-04-16T08:11:15.044 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1341 16 1935 1919 0.0894272 0 - 0.363217 2024-04-16T08:11:16.045 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1342 16 1935 1919 0.0893605 0 - 0.363217 2024-04-16T08:11:17.045 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1343 16 1935 1919 0.089294 0 - 0.363217 2024-04-16T08:11:18.045 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1344 16 1935 1919 0.0892276 0 - 0.363217 2024-04-16T08:11:19.045 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1345 16 1935 1919 0.0891612 0 - 0.363217 2024-04-16T08:11:20.045 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1346 16 1935 1919 0.089095 0 - 0.363217 2024-04-16T08:11:21.045 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1347 16 1935 1919 0.0890288 0 - 0.363217 2024-04-16T08:11:22.045 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1348 16 1935 1919 0.0889628 0 - 0.363217 2024-04-16T08:11:23.046 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1349 16 1935 1919 0.0888968 0 - 0.363217 2024-04-16T08:11:24.046 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1350 16 1935 1919 0.088831 0 - 0.363217 2024-04-16T08:11:25.046 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1351 16 1935 1919 0.0887652 0 - 0.363217 2024-04-16T08:11:26.046 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1352 16 1935 1919 0.0886996 0 - 0.363217 2024-04-16T08:11:27.046 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1353 16 1935 1919 0.088634 0 - 0.363217 2024-04-16T08:11:28.046 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1354 16 1935 1919 0.0885686 0 - 0.363217 2024-04-16T08:11:29.046 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1355 16 1935 1919 0.0885032 0 - 0.363217 2024-04-16T08:11:30.046 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1356 16 1935 1919 0.0884379 0 - 0.363217 2024-04-16T08:11:31.046 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1357 16 1935 1919 0.0883728 0 - 0.363217 2024-04-16T08:11:32.047 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1358 16 1935 1919 0.0883077 0 - 0.363217 2024-04-16T08:11:33.047 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1359 16 1935 1919 0.0882427 0 - 0.363217 2024-04-16T08:11:34.047 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:11:34.048865+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:11:34.047 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:11:34.047 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1360 16 1935 1919 0.0881778 0 - 0.363217 2024-04-16T08:11:35.047 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1361 16 1935 1919 0.088113 0 - 0.363217 2024-04-16T08:11:36.047 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1362 16 1935 1919 0.0880483 0 - 0.363217 2024-04-16T08:11:37.047 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1363 16 1935 1919 0.0879837 0 - 0.363217 2024-04-16T08:11:38.048 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1364 16 1935 1919 0.0879192 0 - 0.363217 2024-04-16T08:11:39.048 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1365 16 1935 1919 0.0878548 0 - 0.363217 2024-04-16T08:11:40.048 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1366 16 1935 1919 0.0877905 0 - 0.363217 2024-04-16T08:11:41.048 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1367 16 1935 1919 0.0877263 0 - 0.363217 2024-04-16T08:11:42.048 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1368 16 1935 1919 0.0876622 0 - 0.363217 2024-04-16T08:11:43.048 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1369 16 1935 1919 0.0875981 0 - 0.363217 2024-04-16T08:11:44.048 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1370 16 1935 1919 0.0875342 0 - 0.363217 2024-04-16T08:11:45.049 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1371 16 1935 1919 0.0874703 0 - 0.363217 2024-04-16T08:11:46.049 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1372 16 1935 1919 0.0874066 0 - 0.363217 2024-04-16T08:11:47.049 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1373 16 1935 1919 0.0873429 0 - 0.363217 2024-04-16T08:11:48.049 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1374 16 1935 1919 0.0872793 0 - 0.363217 2024-04-16T08:11:49.049 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1375 16 1935 1919 0.0872159 0 - 0.363217 2024-04-16T08:11:50.049 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1376 16 1935 1919 0.0871525 0 - 0.363217 2024-04-16T08:11:51.049 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1377 16 1935 1919 0.0870892 0 - 0.363217 2024-04-16T08:11:52.049 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1378 16 1935 1919 0.087026 0 - 0.363217 2024-04-16T08:11:53.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1379 16 1935 1919 0.0869629 0 - 0.363217 2024-04-16T08:11:54.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:11:54.051570+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:11:54.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:11:54.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1380 16 1935 1919 0.0868999 0 - 0.363217 2024-04-16T08:11:55.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1381 16 1935 1919 0.0868369 0 - 0.363217 2024-04-16T08:11:56.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1382 16 1935 1919 0.0867741 0 - 0.363217 2024-04-16T08:11:57.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1383 16 1935 1919 0.0867114 0 - 0.363217 2024-04-16T08:11:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:11:57.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:11:58.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1384 16 1935 1919 0.0866487 0 - 0.363217 2024-04-16T08:11:59.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1385 16 1935 1919 0.0865862 0 - 0.363217 2024-04-16T08:11:59.202 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T08:11:59.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:11:59.323+0000 7fe6e4f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:12:00.050 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1386 16 1935 1919 0.0865237 0 - 0.363217 2024-04-16T08:12:01.051 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1387 16 1935 1919 0.0864613 0 - 0.363217 2024-04-16T08:12:02.051 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1388 16 1935 1919 0.086399 0 - 0.363217 2024-04-16T08:12:03.051 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1389 16 1935 1919 0.0863368 0 - 0.363217 2024-04-16T08:12:04.051 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1390 16 1935 1919 0.0862747 0 - 0.363217 2024-04-16T08:12:05.051 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1391 16 1935 1919 0.0862127 0 - 0.363217 2024-04-16T08:12:06.051 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1392 16 1935 1919 0.0861507 0 - 0.363217 2024-04-16T08:12:07.051 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1393 16 1935 1919 0.0860889 0 - 0.363217 2024-04-16T08:12:08.052 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1394 16 1935 1919 0.0860271 0 - 0.363217 2024-04-16T08:12:09.052 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1395 16 1935 1919 0.0859655 0 - 0.363217 2024-04-16T08:12:10.052 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1396 16 1935 1919 0.0859039 0 - 0.363217 2024-04-16T08:12:11.052 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1397 16 1935 1919 0.0858424 0 - 0.363217 2024-04-16T08:12:12.052 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1398 16 1935 1919 0.085781 0 - 0.363217 2024-04-16T08:12:13.052 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1399 16 1935 1919 0.0857197 0 - 0.363217 2024-04-16T08:12:14.052 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:12:14.054319+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:12:14.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:12:14.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1400 16 1935 1919 0.0856584 0 - 0.363217 2024-04-16T08:12:15.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1401 16 1935 1919 0.0855973 0 - 0.363217 2024-04-16T08:12:16.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1402 16 1935 1919 0.0855362 0 - 0.363217 2024-04-16T08:12:17.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1403 16 1935 1919 0.0854753 0 - 0.363217 2024-04-16T08:12:18.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1404 16 1935 1919 0.0854144 0 - 0.363217 2024-04-16T08:12:19.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1405 16 1935 1919 0.0853536 0 - 0.363217 2024-04-16T08:12:20.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1406 16 1935 1919 0.0852929 0 - 0.363217 2024-04-16T08:12:21.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1407 16 1935 1919 0.0852323 0 - 0.363217 2024-04-16T08:12:22.053 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1408 16 1935 1919 0.0851717 0 - 0.363217 2024-04-16T08:12:23.054 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1409 16 1935 1919 0.0851113 0 - 0.363217 2024-04-16T08:12:24.054 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1410 16 1935 1919 0.0850509 0 - 0.363217 2024-04-16T08:12:25.054 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1411 16 1935 1919 0.0849907 0 - 0.363217 2024-04-16T08:12:26.054 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1412 16 1935 1919 0.0849305 0 - 0.363217 2024-04-16T08:12:27.054 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1413 16 1935 1919 0.0848704 0 - 0.363217 2024-04-16T08:12:28.054 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1414 16 1935 1919 0.0848103 0 - 0.363217 2024-04-16T08:12:29.054 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1415 16 1935 1919 0.0847504 0 - 0.363217 2024-04-16T08:12:30.055 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1416 16 1935 1919 0.0846905 0 - 0.363217 2024-04-16T08:12:31.055 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1417 16 1935 1919 0.0846308 0 - 0.363217 2024-04-16T08:12:32.055 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1418 16 1935 1919 0.0845711 0 - 0.363217 2024-04-16T08:12:33.055 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1419 16 1935 1919 0.0845115 0 - 0.363217 2024-04-16T08:12:34.055 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:12:34.056989+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:12:34.055 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:12:34.055 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1420 16 1935 1919 0.084452 0 - 0.363217 2024-04-16T08:12:35.055 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1421 16 1935 1919 0.0843925 0 - 0.363217 2024-04-16T08:12:36.055 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1422 16 1935 1919 0.0843332 0 - 0.363217 2024-04-16T08:12:37.056 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1423 16 1935 1919 0.0842739 0 - 0.363217 2024-04-16T08:12:38.056 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1424 16 1935 1919 0.0842148 0 - 0.363217 2024-04-16T08:12:39.056 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1425 16 1935 1919 0.0841557 0 - 0.363217 2024-04-16T08:12:40.056 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1426 16 1935 1919 0.0840966 0 - 0.363217 2024-04-16T08:12:41.057 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1427 16 1935 1919 0.0840377 0 - 0.363217 2024-04-16T08:12:42.056 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1428 16 1935 1919 0.0839789 0 - 0.363217 2024-04-16T08:12:43.056 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1429 16 1935 1919 0.0839201 0 - 0.363217 2024-04-16T08:12:44.056 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1430 16 1935 1919 0.0838614 0 - 0.363217 2024-04-16T08:12:45.057 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1431 16 1935 1919 0.0838028 0 - 0.363217 2024-04-16T08:12:46.057 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1432 16 1935 1919 0.0837443 0 - 0.363217 2024-04-16T08:12:47.057 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1433 16 1935 1919 0.0836858 0 - 0.363217 2024-04-16T08:12:48.057 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1434 16 1935 1919 0.0836275 0 - 0.363217 2024-04-16T08:12:49.057 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1435 16 1935 1919 0.0835692 0 - 0.363217 2024-04-16T08:12:50.057 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1436 16 1935 1919 0.083511 0 - 0.363217 2024-04-16T08:12:51.058 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1437 16 1935 1919 0.0834529 0 - 0.363217 2024-04-16T08:12:52.057 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1438 16 1935 1919 0.0833949 0 - 0.363217 2024-04-16T08:12:53.058 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1439 16 1935 1919 0.0833369 0 - 0.363217 2024-04-16T08:12:54.058 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:12:54.059768+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:12:54.058 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:12:54.058 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1440 16 1935 1919 0.083279 0 - 0.363217 2024-04-16T08:12:55.058 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1441 16 1935 1919 0.0832212 0 - 0.363217 2024-04-16T08:12:56.058 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1442 16 1935 1919 0.0831635 0 - 0.363217 2024-04-16T08:12:57.058 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1443 16 1935 1919 0.0831059 0 - 0.363217 2024-04-16T08:12:58.058 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1444 16 1935 1919 0.0830483 0 - 0.363217 2024-04-16T08:12:59.059 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1445 16 1935 1919 0.0829909 0 - 0.363217 2024-04-16T08:13:00.060 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1446 16 1935 1919 0.0829335 0 - 0.363217 2024-04-16T08:13:01.059 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1447 16 1935 1919 0.0828762 0 - 0.363217 2024-04-16T08:13:02.059 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1448 16 1935 1919 0.0828189 0 - 0.363217 2024-04-16T08:13:03.059 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1449 16 1935 1919 0.0827618 0 - 0.363217 2024-04-16T08:13:04.059 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1450 16 1935 1919 0.0827047 0 - 0.363217 2024-04-16T08:13:05.060 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1451 16 1935 1919 0.0826477 0 - 0.363217 2024-04-16T08:13:06.059 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1452 16 1935 1919 0.0825908 0 - 0.363217 2024-04-16T08:13:07.059 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1453 16 1935 1919 0.0825339 0 - 0.363217 2024-04-16T08:13:08.060 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1454 16 1935 1919 0.0824772 0 - 0.363217 2024-04-16T08:13:09.060 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1455 16 1935 1919 0.0824205 0 - 0.363217 2024-04-16T08:13:10.060 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1456 16 1935 1919 0.0823639 0 - 0.363217 2024-04-16T08:13:11.060 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1457 16 1935 1919 0.0823073 0 - 0.363217 2024-04-16T08:13:12.060 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1458 16 1935 1919 0.0822509 0 - 0.363217 2024-04-16T08:13:13.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1459 16 1935 1919 0.0821945 0 - 0.363217 2024-04-16T08:13:14.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:13:14.062382+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:13:14.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:13:14.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1460 16 1935 1919 0.0821382 0 - 0.363217 2024-04-16T08:13:15.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1461 16 1935 1919 0.082082 0 - 0.363217 2024-04-16T08:13:16.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1462 16 1935 1919 0.0820259 0 - 0.363217 2024-04-16T08:13:17.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1463 16 1935 1919 0.0819698 0 - 0.363217 2024-04-16T08:13:18.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1464 16 1935 1919 0.0819138 0 - 0.363217 2024-04-16T08:13:19.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1465 16 1935 1919 0.0818579 0 - 0.363217 2024-04-16T08:13:20.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1466 16 1935 1919 0.081802 0 - 0.363217 2024-04-16T08:13:21.061 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1467 16 1935 1919 0.0817463 0 - 0.363217 2024-04-16T08:13:22.062 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1468 16 1935 1919 0.0816906 0 - 0.363217 2024-04-16T08:13:23.062 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1469 16 1935 1919 0.081635 0 - 0.363217 2024-04-16T08:13:24.062 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1470 16 1935 1919 0.0815795 0 - 0.363217 2024-04-16T08:13:25.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1471 16 1935 1919 0.081524 0 - 0.363217 2024-04-16T08:13:26.062 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1472 16 1935 1919 0.0814686 0 - 0.363217 2024-04-16T08:13:27.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1473 16 1935 1919 0.0814133 0 - 0.363217 2024-04-16T08:13:28.062 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1474 16 1935 1919 0.0813581 0 - 0.363217 2024-04-16T08:13:29.062 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1475 16 1935 1919 0.0813029 0 - 0.363217 2024-04-16T08:13:30.062 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1476 16 1935 1919 0.0812478 0 - 0.363217 2024-04-16T08:13:31.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1477 16 1935 1919 0.0811928 0 - 0.363217 2024-04-16T08:13:32.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1478 16 1935 1919 0.0811379 0 - 0.363217 2024-04-16T08:13:33.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1479 16 1935 1919 0.081083 0 - 0.363217 2024-04-16T08:13:34.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:13:34.065055+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:13:34.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:13:34.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1480 16 1935 1919 0.0810282 0 - 0.363217 2024-04-16T08:13:35.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1481 16 1935 1919 0.0809735 0 - 0.363217 2024-04-16T08:13:36.063 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1482 16 1935 1919 0.0809189 0 - 0.363217 2024-04-16T08:13:37.064 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1483 16 1935 1919 0.0808643 0 - 0.363217 2024-04-16T08:13:38.064 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1484 16 1935 1919 0.0808098 0 - 0.363217 2024-04-16T08:13:39.064 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1485 16 1935 1919 0.0807554 0 - 0.363217 2024-04-16T08:13:40.064 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1486 16 1935 1919 0.0807011 0 - 0.363217 2024-04-16T08:13:41.064 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1487 16 1935 1919 0.0806468 0 - 0.363217 2024-04-16T08:13:42.064 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1488 16 1935 1919 0.0805926 0 - 0.363217 2024-04-16T08:13:43.064 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1489 16 1935 1919 0.0805385 0 - 0.363217 2024-04-16T08:13:44.065 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1490 16 1935 1919 0.0804844 0 - 0.363217 2024-04-16T08:13:45.065 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1491 16 1935 1919 0.0804304 0 - 0.363217 2024-04-16T08:13:46.065 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1492 16 1935 1919 0.0803765 0 - 0.363217 2024-04-16T08:13:47.065 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1493 16 1935 1919 0.0803227 0 - 0.363217 2024-04-16T08:13:48.065 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1494 16 1935 1919 0.0802689 0 - 0.363217 2024-04-16T08:13:49.065 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1495 16 1935 1919 0.0802152 0 - 0.363217 2024-04-16T08:13:50.065 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1496 16 1935 1919 0.0801616 0 - 0.363217 2024-04-16T08:13:51.065 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1497 16 1935 1919 0.0801081 0 - 0.363217 2024-04-16T08:13:52.065 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1498 16 1935 1919 0.0800546 0 - 0.363217 2024-04-16T08:13:53.066 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1499 16 1935 1919 0.0800012 0 - 0.363217 2024-04-16T08:13:54.066 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:13:54.067773+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:13:54.066 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:13:54.066 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1500 16 1935 1919 0.0799479 0 - 0.363217 2024-04-16T08:13:55.067 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1501 16 1935 1919 0.0798946 0 - 0.363217 2024-04-16T08:13:56.067 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1502 16 1935 1919 0.0798414 0 - 0.363217 2024-04-16T08:13:57.066 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1503 16 1935 1919 0.0797883 0 - 0.363217 2024-04-16T08:13:58.066 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1504 16 1935 1919 0.0797352 0 - 0.363217 2024-04-16T08:13:59.066 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1505 16 1935 1919 0.0796822 0 - 0.363217 2024-04-16T08:13:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:13:59.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:14:00.067 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1506 16 1935 1919 0.0796293 0 - 0.363217 2024-04-16T08:14:01.067 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1507 16 1935 1919 0.0795765 0 - 0.363217 2024-04-16T08:14:01.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:14:01.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:14:01.363+0000 7f7c0b59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:14:02.067 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1508 16 1935 1919 0.0795237 0 - 0.363217 2024-04-16T08:14:03.067 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1509 16 1935 1919 0.079471 0 - 0.363217 2024-04-16T08:14:04.067 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1510 16 1935 1919 0.0794184 0 - 0.363217 2024-04-16T08:14:05.067 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1511 16 1935 1919 0.0793658 0 - 0.363217 2024-04-16T08:14:06.068 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1512 16 1935 1919 0.0793133 0 - 0.363217 2024-04-16T08:14:07.068 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1513 16 1935 1919 0.0792609 0 - 0.363217 2024-04-16T08:14:08.068 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1514 16 1935 1919 0.0792086 0 - 0.363217 2024-04-16T08:14:09.068 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1515 16 1935 1919 0.0791563 0 - 0.363217 2024-04-16T08:14:10.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1516 16 1935 1919 0.0791041 0 - 0.363217 2024-04-16T08:14:11.068 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1517 16 1935 1919 0.0790519 0 - 0.363217 2024-04-16T08:14:12.068 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1518 16 1935 1919 0.0789999 0 - 0.363217 2024-04-16T08:14:13.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1519 16 1935 1919 0.0789478 0 - 0.363217 2024-04-16T08:14:14.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:14:14.070528+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:14:14.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:14:14.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1520 16 1935 1919 0.0788959 0 - 0.363217 2024-04-16T08:14:15.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1521 16 1935 1919 0.078844 0 - 0.363217 2024-04-16T08:14:16.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1522 16 1935 1919 0.0787922 0 - 0.363217 2024-04-16T08:14:17.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1523 16 1935 1919 0.0787405 0 - 0.363217 2024-04-16T08:14:18.070 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1524 16 1935 1919 0.0786888 0 - 0.363217 2024-04-16T08:14:19.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1525 16 1935 1919 0.0786372 0 - 0.363217 2024-04-16T08:14:20.069 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1526 16 1935 1919 0.0785857 0 - 0.363217 2024-04-16T08:14:21.070 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1527 16 1935 1919 0.0785342 0 - 0.363217 2024-04-16T08:14:22.070 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1528 16 1935 1919 0.0784828 0 - 0.363217 2024-04-16T08:14:23.070 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1529 16 1935 1919 0.0784315 0 - 0.363217 2024-04-16T08:14:24.070 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1530 16 1935 1919 0.0783802 0 - 0.363217 2024-04-16T08:14:25.070 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1531 16 1935 1919 0.078329 0 - 0.363217 2024-04-16T08:14:26.070 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1532 16 1935 1919 0.0782779 0 - 0.363217 2024-04-16T08:14:27.070 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1533 16 1935 1919 0.0782269 0 - 0.363217 2024-04-16T08:14:28.071 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1534 16 1935 1919 0.0781759 0 - 0.363217 2024-04-16T08:14:29.071 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1535 16 1935 1919 0.0781249 0 - 0.363217 2024-04-16T08:14:30.071 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1536 16 1935 1919 0.0780741 0 - 0.363217 2024-04-16T08:14:31.071 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1537 16 1935 1919 0.0780233 0 - 0.363217 2024-04-16T08:14:32.071 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1538 16 1935 1919 0.0779725 0 - 0.363217 2024-04-16T08:14:33.071 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1539 16 1935 1919 0.0779219 0 - 0.363217 2024-04-16T08:14:34.071 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:14:34.073278+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:14:34.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:14:34.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1540 16 1935 1919 0.0778713 0 - 0.363217 2024-04-16T08:14:35.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1541 16 1935 1919 0.0778207 0 - 0.363217 2024-04-16T08:14:36.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1542 16 1935 1919 0.0777703 0 - 0.363217 2024-04-16T08:14:37.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1543 16 1935 1919 0.0777199 0 - 0.363217 2024-04-16T08:14:38.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1544 16 1935 1919 0.0776695 0 - 0.363217 2024-04-16T08:14:39.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1545 16 1935 1919 0.0776193 0 - 0.363217 2024-04-16T08:14:40.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1546 16 1935 1919 0.0775691 0 - 0.363217 2024-04-16T08:14:41.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1547 16 1935 1919 0.0775189 0 - 0.363217 2024-04-16T08:14:42.072 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1548 16 1935 1919 0.0774688 0 - 0.363217 2024-04-16T08:14:43.073 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1549 16 1935 1919 0.0774188 0 - 0.363217 2024-04-16T08:14:44.073 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1550 16 1935 1919 0.0773689 0 - 0.363217 2024-04-16T08:14:45.073 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1551 16 1935 1919 0.077319 0 - 0.363217 2024-04-16T08:14:46.073 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1552 16 1935 1919 0.0772692 0 - 0.363217 2024-04-16T08:14:47.073 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1553 16 1935 1919 0.0772194 0 - 0.363217 2024-04-16T08:14:48.073 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1554 16 1935 1919 0.0771697 0 - 0.363217 2024-04-16T08:14:49.073 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1555 16 1935 1919 0.0771201 0 - 0.363217 2024-04-16T08:14:50.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1556 16 1935 1919 0.0770705 0 - 0.363217 2024-04-16T08:14:51.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1557 16 1935 1919 0.077021 0 - 0.363217 2024-04-16T08:14:52.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1558 16 1935 1919 0.0769716 0 - 0.363217 2024-04-16T08:14:53.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1559 16 1935 1919 0.0769222 0 - 0.363217 2024-04-16T08:14:54.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:14:54.076036+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:14:54.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:14:54.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1560 16 1935 1919 0.0768729 0 - 0.363217 2024-04-16T08:14:55.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1561 16 1935 1919 0.0768237 0 - 0.363217 2024-04-16T08:14:56.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1562 16 1935 1919 0.0767745 0 - 0.363217 2024-04-16T08:14:57.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1563 16 1935 1919 0.0767254 0 - 0.363217 2024-04-16T08:14:58.074 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1564 16 1935 1919 0.0766763 0 - 0.363217 2024-04-16T08:14:59.075 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1565 16 1935 1919 0.0766273 0 - 0.363217 2024-04-16T08:15:00.075 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1566 16 1935 1919 0.0765784 0 - 0.363217 2024-04-16T08:15:01.075 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1567 16 1935 1919 0.0765295 0 - 0.363217 2024-04-16T08:15:02.075 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1568 16 1935 1919 0.0764807 0 - 0.363217 2024-04-16T08:15:03.076 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1569 16 1935 1919 0.076432 0 - 0.363217 2024-04-16T08:15:04.075 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1570 16 1935 1919 0.0763833 0 - 0.363217 2024-04-16T08:15:05.076 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1571 16 1935 1919 0.0763347 0 - 0.363217 2024-04-16T08:15:06.076 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1572 16 1935 1919 0.0762861 0 - 0.363217 2024-04-16T08:15:07.076 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1573 16 1935 1919 0.0762376 0 - 0.363217 2024-04-16T08:15:08.076 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1574 16 1935 1919 0.0761892 0 - 0.363217 2024-04-16T08:15:09.076 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1575 16 1935 1919 0.0761408 0 - 0.363217 2024-04-16T08:15:10.076 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1576 16 1935 1919 0.0760925 0 - 0.363217 2024-04-16T08:15:11.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1577 16 1935 1919 0.0760442 0 - 0.363217 2024-04-16T08:15:12.076 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1578 16 1935 1919 0.075996 0 - 0.363217 2024-04-16T08:15:13.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1579 16 1935 1919 0.0759479 0 - 0.363217 2024-04-16T08:15:14.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:15:14.078709+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:15:14.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:15:14.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1580 16 1935 1919 0.0758998 0 - 0.363217 2024-04-16T08:15:15.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1581 16 1935 1919 0.0758518 0 - 0.363217 2024-04-16T08:15:16.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1582 16 1935 1919 0.0758039 0 - 0.363217 2024-04-16T08:15:17.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1583 16 1935 1919 0.075756 0 - 0.363217 2024-04-16T08:15:18.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1584 16 1935 1919 0.0757082 0 - 0.363217 2024-04-16T08:15:19.077 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1585 16 1935 1919 0.0756604 0 - 0.363217 2024-04-16T08:15:20.078 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1586 16 1935 1919 0.0756127 0 - 0.363217 2024-04-16T08:15:21.078 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1587 16 1935 1919 0.0755651 0 - 0.363217 2024-04-16T08:15:22.078 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1588 16 1935 1919 0.0755175 0 - 0.363217 2024-04-16T08:15:23.078 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1589 16 1935 1919 0.0754699 0 - 0.363217 2024-04-16T08:15:24.078 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1590 16 1935 1919 0.0754225 0 - 0.363217 2024-04-16T08:15:25.078 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1591 16 1935 1919 0.0753751 0 - 0.363217 2024-04-16T08:15:26.078 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1592 16 1935 1919 0.0753277 0 - 0.363217 2024-04-16T08:15:27.079 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1593 16 1935 1919 0.0752804 0 - 0.363217 2024-04-16T08:15:28.079 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1594 16 1935 1919 0.0752332 0 - 0.363217 2024-04-16T08:15:29.079 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1595 16 1935 1919 0.075186 0 - 0.363217 2024-04-16T08:15:30.079 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1596 16 1935 1919 0.0751389 0 - 0.363217 2024-04-16T08:15:31.079 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1597 16 1935 1919 0.0750919 0 - 0.363217 2024-04-16T08:15:32.079 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1598 16 1935 1919 0.0750449 0 - 0.363217 2024-04-16T08:15:33.079 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1599 16 1935 1919 0.074998 0 - 0.363217 2024-04-16T08:15:34.079 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:15:34.081418+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:15:34.080 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:15:34.080 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1600 16 1935 1919 0.0749511 0 - 0.363217 2024-04-16T08:15:35.080 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1601 16 1935 1919 0.0749043 0 - 0.363217 2024-04-16T08:15:36.080 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1602 16 1935 1919 0.0748575 0 - 0.363217 2024-04-16T08:15:37.080 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1603 16 1935 1919 0.0748108 0 - 0.363217 2024-04-16T08:15:38.080 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1604 16 1935 1919 0.0747642 0 - 0.363217 2024-04-16T08:15:39.080 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1605 16 1935 1919 0.0747176 0 - 0.363217 2024-04-16T08:15:40.080 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1606 16 1935 1919 0.0746711 0 - 0.363217 2024-04-16T08:15:41.081 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1607 16 1935 1919 0.0746246 0 - 0.363217 2024-04-16T08:15:42.081 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1608 16 1935 1919 0.0745782 0 - 0.363217 2024-04-16T08:15:43.081 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1609 16 1935 1919 0.0745318 0 - 0.363217 2024-04-16T08:15:44.081 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1610 16 1935 1919 0.0744856 0 - 0.363217 2024-04-16T08:15:45.081 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1611 16 1935 1919 0.0744393 0 - 0.363217 2024-04-16T08:15:46.081 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1612 16 1935 1919 0.0743931 0 - 0.363217 2024-04-16T08:15:47.081 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1613 16 1935 1919 0.074347 0 - 0.363217 2024-04-16T08:15:48.081 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1614 16 1935 1919 0.074301 0 - 0.363217 2024-04-16T08:15:49.082 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1615 16 1935 1919 0.0742549 0 - 0.363217 2024-04-16T08:15:50.082 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1616 16 1935 1919 0.074209 0 - 0.363217 2024-04-16T08:15:51.082 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1617 16 1935 1919 0.0741631 0 - 0.363217 2024-04-16T08:15:52.082 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1618 16 1935 1919 0.0741173 0 - 0.363217 2024-04-16T08:15:53.082 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1619 16 1935 1919 0.0740715 0 - 0.363217 2024-04-16T08:15:54.082 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:15:54.084169+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:15:54.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:15:54.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1620 16 1935 1919 0.0740258 0 - 0.363217 2024-04-16T08:15:55.082 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1621 16 1935 1919 0.0739801 0 - 0.363217 2024-04-16T08:15:56.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1622 16 1935 1919 0.0739345 0 - 0.363217 2024-04-16T08:15:57.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1623 16 1935 1919 0.0738889 0 - 0.363217 2024-04-16T08:15:58.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1624 16 1935 1919 0.0738434 0 - 0.363217 2024-04-16T08:15:59.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1625 16 1935 1919 0.073798 0 - 0.363217 2024-04-16T08:16:00.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1626 16 1935 1919 0.0737526 0 - 0.363217 2024-04-16T08:16:01.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1627 16 1935 1919 0.0737073 0 - 0.363217 2024-04-16T08:16:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:16:01.283 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:16:02.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1628 16 1935 1919 0.073662 0 - 0.363217 2024-04-16T08:16:03.083 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1629 16 1935 1919 0.0736168 0 - 0.363217 2024-04-16T08:16:03.284 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T08:16:03.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:16:03.406+0000 7fad1ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:16:04.084 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1630 16 1935 1919 0.0735716 0 - 0.363217 2024-04-16T08:16:05.084 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1631 16 1935 1919 0.0735265 0 - 0.363217 2024-04-16T08:16:06.084 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1632 16 1935 1919 0.0734815 0 - 0.363217 2024-04-16T08:16:07.084 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1633 16 1935 1919 0.0734365 0 - 0.363217 2024-04-16T08:16:08.084 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1634 16 1935 1919 0.0733915 0 - 0.363217 2024-04-16T08:16:09.084 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1635 16 1935 1919 0.0733466 0 - 0.363217 2024-04-16T08:16:10.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1636 16 1935 1919 0.0733018 0 - 0.363217 2024-04-16T08:16:11.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1637 16 1935 1919 0.073257 0 - 0.363217 2024-04-16T08:16:12.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1638 16 1935 1919 0.0732123 0 - 0.363217 2024-04-16T08:16:13.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1639 16 1935 1919 0.0731676 0 - 0.363217 2024-04-16T08:16:14.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:16:14.086968+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:16:14.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:16:14.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1640 16 1935 1919 0.073123 0 - 0.363217 2024-04-16T08:16:15.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1641 16 1935 1919 0.0730784 0 - 0.363217 2024-04-16T08:16:16.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1642 16 1935 1919 0.0730339 0 - 0.363217 2024-04-16T08:16:17.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1643 16 1935 1919 0.0729895 0 - 0.363217 2024-04-16T08:16:18.085 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1644 16 1935 1919 0.0729451 0 - 0.363217 2024-04-16T08:16:19.086 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1645 16 1935 1919 0.0729007 0 - 0.363217 2024-04-16T08:16:20.086 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1646 16 1935 1919 0.0728565 0 - 0.363217 2024-04-16T08:16:21.086 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1647 16 1935 1919 0.0728122 0 - 0.363217 2024-04-16T08:16:22.086 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1648 16 1935 1919 0.072768 0 - 0.363217 2024-04-16T08:16:23.086 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1649 16 1935 1919 0.0727239 0 - 0.363217 2024-04-16T08:16:24.086 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1650 16 1935 1919 0.0726798 0 - 0.363217 2024-04-16T08:16:25.086 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1651 16 1935 1919 0.0726358 0 - 0.363217 2024-04-16T08:16:26.086 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1652 16 1935 1919 0.0725918 0 - 0.363217 2024-04-16T08:16:27.087 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1653 16 1935 1919 0.0725479 0 - 0.363217 2024-04-16T08:16:28.087 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1654 16 1935 1919 0.0725041 0 - 0.363217 2024-04-16T08:16:29.087 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1655 16 1935 1919 0.0724603 0 - 0.363217 2024-04-16T08:16:30.087 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1656 16 1935 1919 0.0724165 0 - 0.363217 2024-04-16T08:16:31.087 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1657 16 1935 1919 0.0723728 0 - 0.363217 2024-04-16T08:16:32.087 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1658 16 1935 1919 0.0723291 0 - 0.363217 2024-04-16T08:16:33.087 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1659 16 1935 1919 0.0722855 0 - 0.363217 2024-04-16T08:16:34.088 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:16:34.089552+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:16:34.088 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:16:34.088 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1660 16 1935 1919 0.072242 0 - 0.363217 2024-04-16T08:16:35.088 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1661 16 1935 1919 0.0721985 0 - 0.363217 2024-04-16T08:16:36.088 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1662 16 1935 1919 0.0721551 0 - 0.363217 2024-04-16T08:16:37.088 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1663 16 1935 1919 0.0721117 0 - 0.363217 2024-04-16T08:16:38.088 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1664 16 1935 1919 0.0720683 0 - 0.363217 2024-04-16T08:16:39.088 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1665 16 1935 1919 0.0720251 0 - 0.363217 2024-04-16T08:16:40.088 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1666 16 1935 1919 0.0719818 0 - 0.363217 2024-04-16T08:16:41.089 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1667 16 1935 1919 0.0719386 0 - 0.363217 2024-04-16T08:16:42.089 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1668 16 1935 1919 0.0718955 0 - 0.363217 2024-04-16T08:16:43.089 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1669 16 1935 1919 0.0718524 0 - 0.363217 2024-04-16T08:16:44.089 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1670 16 1935 1919 0.0718094 0 - 0.363217 2024-04-16T08:16:45.089 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1671 16 1935 1919 0.0717664 0 - 0.363217 2024-04-16T08:16:46.089 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1672 16 1935 1919 0.0717235 0 - 0.363217 2024-04-16T08:16:47.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1673 16 1935 1919 0.0716806 0 - 0.363217 2024-04-16T08:16:48.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1674 16 1935 1919 0.0716378 0 - 0.363217 2024-04-16T08:16:49.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1675 16 1935 1919 0.0715951 0 - 0.363217 2024-04-16T08:16:50.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1676 16 1935 1919 0.0715523 0 - 0.363217 2024-04-16T08:16:51.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1677 16 1935 1919 0.0715097 0 - 0.363217 2024-04-16T08:16:52.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1678 16 1935 1919 0.0714671 0 - 0.363217 2024-04-16T08:16:53.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1679 16 1935 1919 0.0714245 0 - 0.363217 2024-04-16T08:16:54.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:16:54.092283+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:16:54.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:16:54.091 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1680 16 1935 1919 0.071382 0 - 0.363217 2024-04-16T08:16:55.090 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1681 16 1935 1919 0.0713395 0 - 0.363217 2024-04-16T08:16:56.091 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1682 16 1935 1919 0.0712971 0 - 0.363217 2024-04-16T08:16:57.091 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1683 16 1935 1919 0.0712547 0 - 0.363217 2024-04-16T08:16:58.091 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1684 16 1935 1919 0.0712124 0 - 0.363217 2024-04-16T08:16:59.091 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1685 16 1935 1919 0.0711702 0 - 0.363217 2024-04-16T08:17:00.091 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1686 16 1935 1919 0.0711279 0 - 0.363217 2024-04-16T08:17:01.091 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1687 16 1935 1919 0.0710858 0 - 0.363217 2024-04-16T08:17:02.091 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1688 16 1935 1919 0.0710437 0 - 0.363217 2024-04-16T08:17:03.091 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1689 16 1935 1919 0.0710016 0 - 0.363217 2024-04-16T08:17:04.092 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1690 16 1935 1919 0.0709596 0 - 0.363217 2024-04-16T08:17:05.092 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1691 16 1935 1919 0.0709176 0 - 0.363217 2024-04-16T08:17:06.092 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1692 16 1935 1919 0.0708757 0 - 0.363217 2024-04-16T08:17:07.092 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1693 16 1935 1919 0.0708338 0 - 0.363217 2024-04-16T08:17:08.092 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1694 16 1935 1919 0.070792 0 - 0.363217 2024-04-16T08:17:09.093 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1695 16 1935 1919 0.0707503 0 - 0.363217 2024-04-16T08:17:10.092 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1696 16 1935 1919 0.0707086 0 - 0.363217 2024-04-16T08:17:11.093 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1697 16 1935 1919 0.0706669 0 - 0.363217 2024-04-16T08:17:12.093 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1698 16 1935 1919 0.0706253 0 - 0.363217 2024-04-16T08:17:13.093 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1699 16 1935 1919 0.0705837 0 - 0.363217 2024-04-16T08:17:14.093 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:17:14.095085+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:17:14.093 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:17:14.093 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1700 16 1935 1919 0.0705422 0 - 0.363217 2024-04-16T08:17:15.093 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1701 16 1935 1919 0.0705007 0 - 0.363217 2024-04-16T08:17:16.093 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1702 16 1935 1919 0.0704593 0 - 0.363217 2024-04-16T08:17:17.094 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1703 16 1935 1919 0.0704179 0 - 0.363217 2024-04-16T08:17:18.094 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1704 16 1935 1919 0.0703766 0 - 0.363217 2024-04-16T08:17:19.094 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1705 16 1935 1919 0.0703353 0 - 0.363217 2024-04-16T08:17:20.094 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1706 16 1935 1919 0.0702941 0 - 0.363217 2024-04-16T08:17:21.094 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1707 16 1935 1919 0.0702529 0 - 0.363217 2024-04-16T08:17:22.094 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1708 16 1935 1919 0.0702118 0 - 0.363217 2024-04-16T08:17:23.094 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1709 16 1935 1919 0.0701707 0 - 0.363217 2024-04-16T08:17:24.095 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1710 16 1935 1919 0.0701297 0 - 0.363217 2024-04-16T08:17:25.095 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1711 16 1935 1919 0.0700887 0 - 0.363217 2024-04-16T08:17:26.095 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1712 16 1935 1919 0.0700477 0 - 0.363217 2024-04-16T08:17:27.095 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1713 16 1935 1919 0.0700068 0 - 0.363217 2024-04-16T08:17:28.095 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1714 16 1935 1919 0.069966 0 - 0.363217 2024-04-16T08:17:29.095 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1715 16 1935 1919 0.0699252 0 - 0.363217 2024-04-16T08:17:30.095 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1716 16 1935 1919 0.0698844 0 - 0.363217 2024-04-16T08:17:31.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1717 16 1935 1919 0.0698437 0 - 0.363217 2024-04-16T08:17:32.095 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1718 16 1935 1919 0.0698031 0 - 0.363217 2024-04-16T08:17:33.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1719 16 1935 1919 0.0697625 0 - 0.363217 2024-04-16T08:17:34.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:17:34.097747+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:17:34.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:17:34.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1720 16 1935 1919 0.0697219 0 - 0.363217 2024-04-16T08:17:35.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1721 16 1935 1919 0.0696814 0 - 0.363217 2024-04-16T08:17:36.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1722 16 1935 1919 0.0696409 0 - 0.363217 2024-04-16T08:17:37.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1723 16 1935 1919 0.0696005 0 - 0.363217 2024-04-16T08:17:38.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1724 16 1935 1919 0.0695602 0 - 0.363217 2024-04-16T08:17:39.096 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1725 16 1935 1919 0.0695198 0 - 0.363217 2024-04-16T08:17:40.097 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1726 16 1935 1919 0.0694796 0 - 0.363217 2024-04-16T08:17:41.097 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1727 16 1935 1919 0.0694393 0 - 0.363217 2024-04-16T08:17:42.097 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1728 16 1935 1919 0.0693991 0 - 0.363217 2024-04-16T08:17:43.097 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1729 16 1935 1919 0.069359 0 - 0.363217 2024-04-16T08:17:44.097 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1730 16 1935 1919 0.0693189 0 - 0.363217 2024-04-16T08:17:45.097 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1731 16 1935 1919 0.0692789 0 - 0.363217 2024-04-16T08:17:46.097 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1732 16 1935 1919 0.0692389 0 - 0.363217 2024-04-16T08:17:47.097 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1733 16 1935 1919 0.0691989 0 - 0.363217 2024-04-16T08:17:48.098 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1734 16 1935 1919 0.069159 0 - 0.363217 2024-04-16T08:17:49.098 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1735 16 1935 1919 0.0691191 0 - 0.363217 2024-04-16T08:17:50.098 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1736 16 1935 1919 0.0690793 0 - 0.363217 2024-04-16T08:17:51.098 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1737 16 1935 1919 0.0690396 0 - 0.363217 2024-04-16T08:17:52.098 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1738 16 1935 1919 0.0689998 0 - 0.363217 2024-04-16T08:17:53.098 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1739 16 1935 1919 0.0689602 0 - 0.363217 2024-04-16T08:17:54.098 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:17:54.100385+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:17:54.099 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:17:54.099 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1740 16 1935 1919 0.0689205 0 - 0.363217 2024-04-16T08:17:55.099 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1741 16 1935 1919 0.0688809 0 - 0.363217 2024-04-16T08:17:56.099 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1742 16 1935 1919 0.0688414 0 - 0.363217 2024-04-16T08:17:57.099 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1743 16 1935 1919 0.0688019 0 - 0.363217 2024-04-16T08:17:58.099 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1744 16 1935 1919 0.0687624 0 - 0.363217 2024-04-16T08:17:59.099 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1745 16 1935 1919 0.068723 0 - 0.363217 2024-04-16T08:18:00.099 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1746 16 1935 1919 0.0686837 0 - 0.363217 2024-04-16T08:18:01.099 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1747 16 1935 1919 0.0686444 0 - 0.363217 2024-04-16T08:18:02.100 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1748 16 1935 1919 0.0686051 0 - 0.363217 2024-04-16T08:18:03.100 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1749 16 1935 1919 0.0685659 0 - 0.363217 2024-04-16T08:18:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:18:03.324 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:18:04.100 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1750 16 1935 1919 0.0685267 0 - 0.363217 2024-04-16T08:18:05.100 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1751 16 1935 1919 0.0684875 0 - 0.363217 2024-04-16T08:18:05.325 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:18:05.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:18:05.446+0000 7fc4feeed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:18:06.100 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1752 16 1935 1919 0.0684485 0 - 0.363217 2024-04-16T08:18:07.101 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1753 16 1935 1919 0.0684094 0 - 0.363217 2024-04-16T08:18:08.101 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1754 16 1935 1919 0.0683704 0 - 0.363217 2024-04-16T08:18:09.101 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1755 16 1935 1919 0.0683314 0 - 0.363217 2024-04-16T08:18:10.101 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1756 16 1935 1919 0.0682925 0 - 0.363217 2024-04-16T08:18:11.101 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1757 16 1935 1919 0.0682537 0 - 0.363217 2024-04-16T08:18:12.101 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1758 16 1935 1919 0.0682148 0 - 0.363217 2024-04-16T08:18:13.101 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1759 16 1935 1919 0.0681761 0 - 0.363217 2024-04-16T08:18:14.101 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:18:14.103328+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:18:14.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:18:14.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1760 16 1935 1919 0.0681373 0 - 0.363217 2024-04-16T08:18:15.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1761 16 1935 1919 0.0680986 0 - 0.363217 2024-04-16T08:18:16.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1762 16 1935 1919 0.06806 0 - 0.363217 2024-04-16T08:18:17.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1763 16 1935 1919 0.0680214 0 - 0.363217 2024-04-16T08:18:18.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1764 16 1935 1919 0.0679828 0 - 0.363217 2024-04-16T08:18:19.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1765 16 1935 1919 0.0679443 0 - 0.363217 2024-04-16T08:18:20.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1766 16 1935 1919 0.0679058 0 - 0.363217 2024-04-16T08:18:21.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1767 16 1935 1919 0.0678674 0 - 0.363217 2024-04-16T08:18:22.102 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1768 16 1935 1919 0.067829 0 - 0.363217 2024-04-16T08:18:23.103 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1769 16 1935 1919 0.0677907 0 - 0.363217 2024-04-16T08:18:24.103 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1770 16 1935 1919 0.0677524 0 - 0.363217 2024-04-16T08:18:25.103 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1771 16 1935 1919 0.0677141 0 - 0.363217 2024-04-16T08:18:26.103 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1772 16 1935 1919 0.0676759 0 - 0.363217 2024-04-16T08:18:27.103 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1773 16 1935 1919 0.0676377 0 - 0.363217 2024-04-16T08:18:28.103 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1774 16 1935 1919 0.0675996 0 - 0.363217 2024-04-16T08:18:29.103 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1775 16 1935 1919 0.0675615 0 - 0.363217 2024-04-16T08:18:30.103 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1776 16 1935 1919 0.0675235 0 - 0.363217 2024-04-16T08:18:31.104 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1777 16 1935 1919 0.0674855 0 - 0.363217 2024-04-16T08:18:32.104 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1778 16 1935 1919 0.0674475 0 - 0.363217 2024-04-16T08:18:33.104 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1779 16 1935 1919 0.0674096 0 - 0.363217 2024-04-16T08:18:34.104 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:18:34.106039+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:18:34.104 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:18:34.104 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1780 16 1935 1919 0.0673717 0 - 0.363217 2024-04-16T08:18:35.104 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1781 16 1935 1919 0.0673339 0 - 0.363217 2024-04-16T08:18:36.104 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1782 16 1935 1919 0.0672961 0 - 0.363217 2024-04-16T08:18:37.104 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1783 16 1935 1919 0.0672584 0 - 0.363217 2024-04-16T08:18:38.105 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1784 16 1935 1919 0.0672207 0 - 0.363217 2024-04-16T08:18:39.105 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1785 16 1935 1919 0.067183 0 - 0.363217 2024-04-16T08:18:40.105 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1786 16 1935 1919 0.0671454 0 - 0.363217 2024-04-16T08:18:41.105 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1787 16 1935 1919 0.0671078 0 - 0.363217 2024-04-16T08:18:42.105 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1788 16 1935 1919 0.0670703 0 - 0.363217 2024-04-16T08:18:43.105 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1789 16 1935 1919 0.0670328 0 - 0.363217 2024-04-16T08:18:44.105 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1790 16 1935 1919 0.0669953 0 - 0.363217 2024-04-16T08:18:45.106 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1791 16 1935 1919 0.0669579 0 - 0.363217 2024-04-16T08:18:46.106 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1792 16 1935 1919 0.0669206 0 - 0.363217 2024-04-16T08:18:47.106 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1793 16 1935 1919 0.0668833 0 - 0.363217 2024-04-16T08:18:48.106 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1794 16 1935 1919 0.066846 0 - 0.363217 2024-04-16T08:18:49.106 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1795 16 1935 1919 0.0668087 0 - 0.363217 2024-04-16T08:18:50.106 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1796 16 1935 1919 0.0667715 0 - 0.363217 2024-04-16T08:18:51.106 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1797 16 1935 1919 0.0667344 0 - 0.363217 2024-04-16T08:18:52.106 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1798 16 1935 1919 0.0666973 0 - 0.363217 2024-04-16T08:18:53.107 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1799 16 1935 1919 0.0666602 0 - 0.363217 2024-04-16T08:18:54.107 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:18:54.108728+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:18:54.107 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:18:54.107 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1800 16 1935 1919 0.0666232 0 - 0.363217 2024-04-16T08:18:55.107 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1801 16 1935 1919 0.0665862 0 - 0.363217 2024-04-16T08:18:56.107 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1802 16 1935 1919 0.0665492 0 - 0.363217 2024-04-16T08:18:57.107 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1803 16 1935 1919 0.0665123 0 - 0.363217 2024-04-16T08:18:58.107 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1804 16 1935 1919 0.0664754 0 - 0.363217 2024-04-16T08:18:59.108 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1805 16 1935 1919 0.0664386 0 - 0.363217 2024-04-16T08:19:00.108 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1806 16 1935 1919 0.0664018 0 - 0.363217 2024-04-16T08:19:01.108 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1807 16 1935 1919 0.0663651 0 - 0.363217 2024-04-16T08:19:02.108 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1808 16 1935 1919 0.0663284 0 - 0.363217 2024-04-16T08:19:03.108 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1809 16 1935 1919 0.0662917 0 - 0.363217 2024-04-16T08:19:04.108 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1810 16 1935 1919 0.0662551 0 - 0.363217 2024-04-16T08:19:05.108 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1811 16 1935 1919 0.0662185 0 - 0.363217 2024-04-16T08:19:06.108 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1812 16 1935 1919 0.0661819 0 - 0.363217 2024-04-16T08:19:07.109 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1813 16 1935 1919 0.0661454 0 - 0.363217 2024-04-16T08:19:08.109 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1814 16 1935 1919 0.066109 0 - 0.363217 2024-04-16T08:19:09.109 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1815 16 1935 1919 0.0660725 0 - 0.363217 2024-04-16T08:19:10.109 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1816 16 1935 1919 0.0660362 0 - 0.363217 2024-04-16T08:19:11.109 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1817 16 1935 1919 0.0659998 0 - 0.363217 2024-04-16T08:19:12.109 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1818 16 1935 1919 0.0659635 0 - 0.363217 2024-04-16T08:19:13.110 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1819 16 1935 1919 0.0659272 0 - 0.363217 2024-04-16T08:19:14.109 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:19:14.111512+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:19:14.110 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:19:14.110 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1820 16 1935 1919 0.065891 0 - 0.363217 2024-04-16T08:19:15.110 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1821 16 1935 1919 0.0658548 0 - 0.363217 2024-04-16T08:19:16.110 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1822 16 1935 1919 0.0658187 0 - 0.363217 2024-04-16T08:19:17.110 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1823 16 1935 1919 0.0657826 0 - 0.363217 2024-04-16T08:19:18.110 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1824 16 1935 1919 0.0657465 0 - 0.363217 2024-04-16T08:19:19.111 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1825 16 1935 1919 0.0657105 0 - 0.363217 2024-04-16T08:19:20.110 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1826 16 1935 1919 0.0656745 0 - 0.363217 2024-04-16T08:19:21.110 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1827 16 1935 1919 0.0656386 0 - 0.363217 2024-04-16T08:19:22.111 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1828 16 1935 1919 0.0656027 0 - 0.363217 2024-04-16T08:19:23.111 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1829 16 1935 1919 0.0655668 0 - 0.363217 2024-04-16T08:19:24.111 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1830 16 1935 1919 0.065531 0 - 0.363217 2024-04-16T08:19:25.111 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1831 16 1935 1919 0.0654952 0 - 0.363217 2024-04-16T08:19:26.111 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1832 16 1935 1919 0.0654594 0 - 0.363217 2024-04-16T08:19:27.111 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1833 16 1935 1919 0.0654237 0 - 0.363217 2024-04-16T08:19:28.111 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1834 16 1935 1919 0.065388 0 - 0.363217 2024-04-16T08:19:29.111 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1835 16 1935 1919 0.0653524 0 - 0.363217 2024-04-16T08:19:30.112 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1836 16 1935 1919 0.0653168 0 - 0.363217 2024-04-16T08:19:31.112 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1837 16 1935 1919 0.0652813 0 - 0.363217 2024-04-16T08:19:32.112 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1838 16 1935 1919 0.0652457 0 - 0.363217 2024-04-16T08:19:33.112 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1839 16 1935 1919 0.0652103 0 - 0.363217 2024-04-16T08:19:34.112 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:19:34.114112+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:19:34.112 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:19:34.112 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1840 16 1935 1919 0.0651748 0 - 0.363217 2024-04-16T08:19:35.112 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1841 16 1935 1919 0.0651394 0 - 0.363217 2024-04-16T08:19:36.112 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1842 16 1935 1919 0.0651041 0 - 0.363217 2024-04-16T08:19:37.113 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1843 16 1935 1919 0.0650687 0 - 0.363217 2024-04-16T08:19:38.113 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1844 16 1935 1919 0.0650334 0 - 0.363217 2024-04-16T08:19:39.113 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1845 16 1935 1919 0.0649982 0 - 0.363217 2024-04-16T08:19:40.113 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1846 16 1935 1919 0.064963 0 - 0.363217 2024-04-16T08:19:41.113 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1847 16 1935 1919 0.0649278 0 - 0.363217 2024-04-16T08:19:42.113 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1848 16 1935 1919 0.0648927 0 - 0.363217 2024-04-16T08:19:43.113 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1849 16 1935 1919 0.0648576 0 - 0.363217 2024-04-16T08:19:44.114 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1850 16 1935 1919 0.0648225 0 - 0.363217 2024-04-16T08:19:45.114 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1851 16 1935 1919 0.0647875 0 - 0.363217 2024-04-16T08:19:46.114 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1852 16 1935 1919 0.0647525 0 - 0.363217 2024-04-16T08:19:47.114 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1853 16 1935 1919 0.0647176 0 - 0.363217 2024-04-16T08:19:48.114 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1854 16 1935 1919 0.0646827 0 - 0.363217 2024-04-16T08:19:49.114 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1855 16 1935 1919 0.0646478 0 - 0.363217 2024-04-16T08:19:50.114 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1856 16 1935 1919 0.064613 0 - 0.363217 2024-04-16T08:19:51.114 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1857 16 1935 1919 0.0645782 0 - 0.363217 2024-04-16T08:19:52.114 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1858 16 1935 1919 0.0645434 0 - 0.363217 2024-04-16T08:19:53.115 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1859 16 1935 1919 0.0645087 0 - 0.363217 2024-04-16T08:19:54.115 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:19:54.116701+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:19:54.115 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:19:54.115 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1860 16 1935 1919 0.064474 0 - 0.363217 2024-04-16T08:19:55.115 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1861 16 1935 1919 0.0644394 0 - 0.363217 2024-04-16T08:19:56.115 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1862 16 1935 1919 0.0644048 0 - 0.363217 2024-04-16T08:19:57.115 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1863 16 1935 1919 0.0643702 0 - 0.363217 2024-04-16T08:19:58.115 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1864 16 1935 1919 0.0643357 0 - 0.363217 2024-04-16T08:19:59.115 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1865 16 1935 1919 0.0643012 0 - 0.363217 2024-04-16T08:20:00.116 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1866 16 1935 1919 0.0642667 0 - 0.363217 2024-04-16T08:20:01.116 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1867 16 1935 1919 0.0642323 0 - 0.363217 2024-04-16T08:20:02.116 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1868 16 1935 1919 0.0641979 0 - 0.363217 2024-04-16T08:20:03.116 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1869 16 1935 1919 0.0641635 0 - 0.363217 2024-04-16T08:20:04.116 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1870 16 1935 1919 0.0641292 0 - 0.363217 2024-04-16T08:20:05.116 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1871 16 1935 1919 0.064095 0 - 0.363217 2024-04-16T08:20:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:20:05.365 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:20:06.116 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1872 16 1935 1919 0.0640607 0 - 0.363217 2024-04-16T08:20:07.116 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1873 16 1935 1919 0.0640265 0 - 0.363217 2024-04-16T08:20:07.366 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T08:20:07.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:20:07.489+0000 7f249fd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:20:08.116 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1874 16 1935 1919 0.0639924 0 - 0.363217 2024-04-16T08:20:09.117 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1875 16 1935 1919 0.0639582 0 - 0.363217 2024-04-16T08:20:10.117 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1876 16 1935 1919 0.0639241 0 - 0.363217 2024-04-16T08:20:11.117 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1877 16 1935 1919 0.0638901 0 - 0.363217 2024-04-16T08:20:12.117 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1878 16 1935 1919 0.0638561 0 - 0.363217 2024-04-16T08:20:13.117 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1879 16 1935 1919 0.0638221 0 - 0.363217 2024-04-16T08:20:14.117 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:20:14.119271+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:20:14.117 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:20:14.118 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1880 16 1935 1919 0.0637881 0 - 0.363217 2024-04-16T08:20:15.117 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1881 16 1935 1919 0.0637542 0 - 0.363217 2024-04-16T08:20:16.117 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1882 16 1935 1919 0.0637203 0 - 0.363217 2024-04-16T08:20:17.118 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1883 16 1935 1919 0.0636865 0 - 0.363217 2024-04-16T08:20:18.118 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1884 16 1935 1919 0.0636527 0 - 0.363217 2024-04-16T08:20:19.118 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1885 16 1935 1919 0.0636189 0 - 0.363217 2024-04-16T08:20:20.118 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1886 16 1935 1919 0.0635852 0 - 0.363217 2024-04-16T08:20:21.118 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1887 16 1935 1919 0.0635515 0 - 0.363217 2024-04-16T08:20:22.118 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1888 16 1935 1919 0.0635178 0 - 0.363217 2024-04-16T08:20:23.118 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1889 16 1935 1919 0.0634842 0 - 0.363217 2024-04-16T08:20:24.119 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1890 16 1935 1919 0.0634506 0 - 0.363217 2024-04-16T08:20:25.119 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1891 16 1935 1919 0.0634171 0 - 0.363217 2024-04-16T08:20:26.119 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1892 16 1935 1919 0.0633836 0 - 0.363217 2024-04-16T08:20:27.119 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1893 16 1935 1919 0.0633501 0 - 0.363217 2024-04-16T08:20:28.119 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1894 16 1935 1919 0.0633166 0 - 0.363217 2024-04-16T08:20:29.119 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1895 16 1935 1919 0.0632832 0 - 0.363217 2024-04-16T08:20:30.119 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1896 16 1935 1919 0.0632498 0 - 0.363217 2024-04-16T08:20:31.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1897 16 1935 1919 0.0632165 0 - 0.363217 2024-04-16T08:20:32.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1898 16 1935 1919 0.0631832 0 - 0.363217 2024-04-16T08:20:33.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1899 16 1935 1919 0.0631499 0 - 0.363217 2024-04-16T08:20:34.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:20:34.121915+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:20:34.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:20:34.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1900 16 1935 1919 0.0631167 0 - 0.363217 2024-04-16T08:20:35.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1901 16 1935 1919 0.0630835 0 - 0.363217 2024-04-16T08:20:36.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1902 16 1935 1919 0.0630503 0 - 0.363217 2024-04-16T08:20:37.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1903 16 1935 1919 0.0630172 0 - 0.363217 2024-04-16T08:20:38.120 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1904 16 1935 1919 0.0629841 0 - 0.363217 2024-04-16T08:20:39.121 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1905 16 1935 1919 0.062951 0 - 0.363217 2024-04-16T08:20:40.121 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1906 16 1935 1919 0.062918 0 - 0.363217 2024-04-16T08:20:41.121 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1907 16 1935 1919 0.062885 0 - 0.363217 2024-04-16T08:20:42.121 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1908 16 1935 1919 0.062852 0 - 0.363217 2024-04-16T08:20:43.121 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1909 16 1935 1919 0.0628191 0 - 0.363217 2024-04-16T08:20:44.121 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1910 16 1935 1919 0.0627862 0 - 0.363217 2024-04-16T08:20:45.121 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1911 16 1935 1919 0.0627534 0 - 0.363217 2024-04-16T08:20:46.121 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1912 16 1935 1919 0.0627205 0 - 0.363217 2024-04-16T08:20:47.122 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1913 16 1935 1919 0.0626878 0 - 0.363217 2024-04-16T08:20:48.122 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1914 16 1935 1919 0.062655 0 - 0.363217 2024-04-16T08:20:49.122 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1915 16 1935 1919 0.0626223 0 - 0.363217 2024-04-16T08:20:50.122 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1916 16 1935 1919 0.0625896 0 - 0.363217 2024-04-16T08:20:51.122 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1917 16 1935 1919 0.062557 0 - 0.363217 2024-04-16T08:20:52.122 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1918 16 1935 1919 0.0625243 0 - 0.363217 2024-04-16T08:20:53.125 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1919 16 1935 1919 0.0624918 0 - 0.363217 2024-04-16T08:20:54.123 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:20:54.124575+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:20:54.123 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:20:54.123 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1920 16 1935 1919 0.0624592 0 - 0.363217 2024-04-16T08:20:55.123 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1921 16 1935 1919 0.0624267 0 - 0.363217 2024-04-16T08:20:56.123 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1922 16 1935 1919 0.0623942 0 - 0.363217 2024-04-16T08:20:57.123 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1923 16 1935 1919 0.0623618 0 - 0.363217 2024-04-16T08:20:58.123 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1924 16 1935 1919 0.0623294 0 - 0.363217 2024-04-16T08:20:59.123 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1925 16 1935 1919 0.062297 0 - 0.363217 2024-04-16T08:21:00.124 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1926 16 1935 1919 0.0622646 0 - 0.363217 2024-04-16T08:21:01.124 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1927 16 1935 1919 0.0622323 0 - 0.363217 2024-04-16T08:21:02.124 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1928 16 1935 1919 0.0622 0 - 0.363217 2024-04-16T08:21:03.124 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1929 16 1935 1919 0.0621678 0 - 0.363217 2024-04-16T08:21:04.124 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1930 16 1935 1919 0.0621356 0 - 0.363217 2024-04-16T08:21:05.124 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1931 16 1935 1919 0.0621034 0 - 0.363217 2024-04-16T08:21:06.124 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1932 16 1935 1919 0.0620713 0 - 0.363217 2024-04-16T08:21:07.124 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1933 16 1935 1919 0.0620391 0 - 0.363217 2024-04-16T08:21:08.125 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1934 16 1935 1919 0.0620071 0 - 0.363217 2024-04-16T08:21:09.125 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1935 16 1935 1919 0.061975 0 - 0.363217 2024-04-16T08:21:10.125 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1936 16 1935 1919 0.061943 0 - 0.363217 2024-04-16T08:21:11.125 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1937 16 1935 1919 0.061911 0 - 0.363217 2024-04-16T08:21:12.125 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1938 16 1935 1919 0.0618791 0 - 0.363217 2024-04-16T08:21:13.125 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1939 16 1935 1919 0.0618472 0 - 0.363217 2024-04-16T08:21:14.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:21:14.127312+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:21:14.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:21:14.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1940 16 1935 1919 0.0618153 0 - 0.363217 2024-04-16T08:21:15.125 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1941 16 1935 1919 0.0617834 0 - 0.363217 2024-04-16T08:21:16.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1942 16 1935 1919 0.0617516 0 - 0.363217 2024-04-16T08:21:17.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1943 16 1935 1919 0.0617198 0 - 0.363217 2024-04-16T08:21:18.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1944 16 1935 1919 0.0616881 0 - 0.363217 2024-04-16T08:21:19.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1945 16 1935 1919 0.0616564 0 - 0.363217 2024-04-16T08:21:20.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1946 16 1935 1919 0.0616247 0 - 0.363217 2024-04-16T08:21:21.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1947 16 1935 1919 0.061593 0 - 0.363217 2024-04-16T08:21:22.126 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1948 16 1935 1919 0.0615614 0 - 0.363217 2024-04-16T08:21:23.127 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1949 16 1935 1919 0.0615298 0 - 0.363217 2024-04-16T08:21:24.127 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1950 16 1935 1919 0.0614983 0 - 0.363217 2024-04-16T08:21:25.127 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1951 16 1935 1919 0.0614668 0 - 0.363217 2024-04-16T08:21:26.127 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1952 16 1935 1919 0.0614353 0 - 0.363217 2024-04-16T08:21:27.127 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1953 16 1935 1919 0.0614038 0 - 0.363217 2024-04-16T08:21:28.127 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1954 16 1935 1919 0.0613724 0 - 0.363217 2024-04-16T08:21:29.127 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1955 16 1935 1919 0.061341 0 - 0.363217 2024-04-16T08:21:30.128 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1956 16 1935 1919 0.0613096 0 - 0.363217 2024-04-16T08:21:31.128 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1957 16 1935 1919 0.0612783 0 - 0.363217 2024-04-16T08:21:32.128 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1958 16 1935 1919 0.061247 0 - 0.363217 2024-04-16T08:21:33.128 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1959 16 1935 1919 0.0612158 0 - 0.363217 2024-04-16T08:21:34.128 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:21:34.130021+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:21:34.128 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:21:34.128 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1960 16 1935 1919 0.0611845 0 - 0.363217 2024-04-16T08:21:35.128 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1961 16 1935 1919 0.0611533 0 - 0.363217 2024-04-16T08:21:36.129 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1962 16 1935 1919 0.0611222 0 - 0.363217 2024-04-16T08:21:37.128 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1963 16 1935 1919 0.061091 0 - 0.363217 2024-04-16T08:21:38.129 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1964 16 1935 1919 0.0610599 0 - 0.363217 2024-04-16T08:21:39.129 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1965 16 1935 1919 0.0610288 0 - 0.363217 2024-04-16T08:21:40.129 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1966 16 1935 1919 0.0609978 0 - 0.363217 2024-04-16T08:21:41.129 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1967 16 1935 1919 0.0609668 0 - 0.363217 2024-04-16T08:21:42.129 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1968 16 1935 1919 0.0609358 0 - 0.363217 2024-04-16T08:21:43.130 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1969 16 1935 1919 0.0609049 0 - 0.363217 2024-04-16T08:21:44.129 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1970 16 1935 1919 0.0608739 0 - 0.363217 2024-04-16T08:21:45.130 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1971 16 1935 1919 0.0608431 0 - 0.363217 2024-04-16T08:21:46.130 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1972 16 1935 1919 0.0608122 0 - 0.363217 2024-04-16T08:21:47.130 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1973 16 1935 1919 0.0607814 0 - 0.363217 2024-04-16T08:21:48.130 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1974 16 1935 1919 0.0607506 0 - 0.363217 2024-04-16T08:21:49.130 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1975 16 1935 1919 0.0607198 0 - 0.363217 2024-04-16T08:21:50.130 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1976 16 1935 1919 0.0606891 0 - 0.363217 2024-04-16T08:21:51.130 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1977 16 1935 1919 0.0606584 0 - 0.363217 2024-04-16T08:21:52.130 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1978 16 1935 1919 0.0606277 0 - 0.363217 2024-04-16T08:21:53.131 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1979 16 1935 1919 0.0605971 0 - 0.363217 2024-04-16T08:21:54.131 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:21:54.132758+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:21:54.131 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:21:54.131 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1980 16 1935 1919 0.0605665 0 - 0.363217 2024-04-16T08:21:55.131 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1981 16 1935 1919 0.0605359 0 - 0.363217 2024-04-16T08:21:56.131 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1982 16 1935 1919 0.0605054 0 - 0.363217 2024-04-16T08:21:57.131 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1983 16 1935 1919 0.0604749 0 - 0.363217 2024-04-16T08:21:58.131 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1984 16 1935 1919 0.0604444 0 - 0.363217 2024-04-16T08:21:59.132 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1985 16 1935 1919 0.0604139 0 - 0.363217 2024-04-16T08:22:00.132 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1986 16 1935 1919 0.0603835 0 - 0.363217 2024-04-16T08:22:01.135 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1987 16 1935 1919 0.0603531 0 - 0.363217 2024-04-16T08:22:02.132 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1988 16 1935 1919 0.0603228 0 - 0.363217 2024-04-16T08:22:03.132 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1989 16 1935 1919 0.0602924 0 - 0.363217 2024-04-16T08:22:04.132 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1990 16 1935 1919 0.0602621 0 - 0.363217 2024-04-16T08:22:05.132 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1991 16 1935 1919 0.0602319 0 - 0.363217 2024-04-16T08:22:06.132 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1992 16 1935 1919 0.0602016 0 - 0.363217 2024-04-16T08:22:07.133 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1993 16 1935 1919 0.0601714 0 - 0.363217 2024-04-16T08:22:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:22:07.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:22:08.133 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1994 16 1935 1919 0.0601413 0 - 0.363217 2024-04-16T08:22:09.133 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1995 16 1935 1919 0.0601111 0 - 0.363217 2024-04-16T08:22:09.408 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:22:09.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:22:09.530+0000 7f33e7f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:22:10.133 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1996 16 1935 1919 0.060081 0 - 0.363217 2024-04-16T08:22:11.133 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1997 16 1935 1919 0.0600509 0 - 0.363217 2024-04-16T08:22:12.133 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1998 16 1935 1919 0.0600208 0 - 0.363217 2024-04-16T08:22:13.133 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 1999 16 1935 1919 0.0599908 0 - 0.363217 2024-04-16T08:22:14.133 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:22:14.135549+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:22:14.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:22:14.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2000 16 1935 1919 0.0599608 0 - 0.363217 2024-04-16T08:22:15.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2001 16 1935 1919 0.0599309 0 - 0.363217 2024-04-16T08:22:16.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2002 16 1935 1919 0.0599009 0 - 0.363217 2024-04-16T08:22:17.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2003 16 1935 1919 0.059871 0 - 0.363217 2024-04-16T08:22:18.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2004 16 1935 1919 0.0598411 0 - 0.363217 2024-04-16T08:22:19.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2005 16 1935 1919 0.0598113 0 - 0.363217 2024-04-16T08:22:20.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2006 16 1935 1919 0.0597815 0 - 0.363217 2024-04-16T08:22:21.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2007 16 1935 1919 0.0597517 0 - 0.363217 2024-04-16T08:22:22.134 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2008 16 1935 1919 0.0597219 0 - 0.363217 2024-04-16T08:22:23.135 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2009 16 1935 1919 0.0596922 0 - 0.363217 2024-04-16T08:22:24.135 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2010 16 1935 1919 0.0596625 0 - 0.363217 2024-04-16T08:22:25.135 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2011 16 1935 1919 0.0596328 0 - 0.363217 2024-04-16T08:22:26.135 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2012 16 1935 1919 0.0596032 0 - 0.363217 2024-04-16T08:22:27.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2013 16 1935 1919 0.0595736 0 - 0.363217 2024-04-16T08:22:28.135 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2014 16 1935 1919 0.059544 0 - 0.363217 2024-04-16T08:22:29.135 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2015 16 1935 1919 0.0595145 0 - 0.363217 2024-04-16T08:22:30.135 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2016 16 1935 1919 0.0594849 0 - 0.363217 2024-04-16T08:22:31.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2017 16 1935 1919 0.0594555 0 - 0.363217 2024-04-16T08:22:32.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2018 16 1935 1919 0.059426 0 - 0.363217 2024-04-16T08:22:33.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2019 16 1935 1919 0.0593966 0 - 0.363217 2024-04-16T08:22:34.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:22:34.138011+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:22:34.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:22:34.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2020 16 1935 1919 0.0593672 0 - 0.363217 2024-04-16T08:22:35.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2021 16 1935 1919 0.0593378 0 - 0.363217 2024-04-16T08:22:36.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2022 16 1935 1919 0.0593084 0 - 0.363217 2024-04-16T08:22:37.138 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2023 16 1935 1919 0.0592791 0 - 0.363217 2024-04-16T08:22:38.136 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2024 16 1935 1919 0.0592498 0 - 0.363217 2024-04-16T08:22:39.137 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2025 16 1935 1919 0.0592206 0 - 0.363217 2024-04-16T08:22:40.137 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2026 16 1935 1919 0.0591913 0 - 0.363217 2024-04-16T08:22:41.137 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2027 16 1935 1919 0.0591621 0 - 0.363217 2024-04-16T08:22:42.137 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2028 16 1935 1919 0.059133 0 - 0.363217 2024-04-16T08:22:43.137 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2029 16 1935 1919 0.0591038 0 - 0.363217 2024-04-16T08:22:44.137 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2030 16 1935 1919 0.0590747 0 - 0.363217 2024-04-16T08:22:45.137 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2031 16 1935 1919 0.0590456 0 - 0.363217 2024-04-16T08:22:46.138 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2032 16 1935 1919 0.0590166 0 - 0.363217 2024-04-16T08:22:47.138 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2033 16 1935 1919 0.0589875 0 - 0.363217 2024-04-16T08:22:48.138 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2034 16 1935 1919 0.0589585 0 - 0.363217 2024-04-16T08:22:49.138 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2035 16 1935 1919 0.0589296 0 - 0.363217 2024-04-16T08:22:50.138 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2036 16 1935 1919 0.0589006 0 - 0.363217 2024-04-16T08:22:51.138 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2037 16 1935 1919 0.0588717 0 - 0.363217 2024-04-16T08:22:52.138 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2038 16 1935 1919 0.0588428 0 - 0.363217 2024-04-16T08:22:53.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2039 16 1935 1919 0.058814 0 - 0.363217 2024-04-16T08:22:54.138 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:22:54.140539+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:22:54.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:22:54.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2040 16 1935 1919 0.0587851 0 - 0.363217 2024-04-16T08:22:55.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2041 16 1935 1919 0.0587563 0 - 0.363217 2024-04-16T08:22:56.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2042 16 1935 1919 0.0587276 0 - 0.363217 2024-04-16T08:22:57.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2043 16 1935 1919 0.0586988 0 - 0.363217 2024-04-16T08:22:58.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2044 16 1935 1919 0.0586701 0 - 0.363217 2024-04-16T08:22:59.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2045 16 1935 1919 0.0586414 0 - 0.363217 2024-04-16T08:23:00.140 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2046 16 1935 1919 0.0586127 0 - 0.363217 2024-04-16T08:23:01.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2047 16 1935 1919 0.0585841 0 - 0.363217 2024-04-16T08:23:02.139 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2048 16 1935 1919 0.0585555 0 - 0.363217 2024-04-16T08:23:03.140 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2049 16 1935 1919 0.0585269 0 - 0.363217 2024-04-16T08:23:04.140 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2050 16 1935 1919 0.0584984 0 - 0.363217 2024-04-16T08:23:05.140 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2051 16 1935 1919 0.0584699 0 - 0.363217 2024-04-16T08:23:06.140 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2052 16 1935 1919 0.0584414 0 - 0.363217 2024-04-16T08:23:07.140 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2053 16 1935 1919 0.0584129 0 - 0.363217 2024-04-16T08:23:08.140 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2054 16 1935 1919 0.0583845 0 - 0.363217 2024-04-16T08:23:09.140 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2055 16 1935 1919 0.058356 0 - 0.363217 2024-04-16T08:23:10.140 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2056 16 1935 1919 0.0583277 0 - 0.363217 2024-04-16T08:23:11.141 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2057 16 1935 1919 0.0582993 0 - 0.363217 2024-04-16T08:23:12.141 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2058 16 1935 1919 0.058271 0 - 0.363217 2024-04-16T08:23:13.141 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2059 16 1935 1919 0.0582427 0 - 0.363217 2024-04-16T08:23:14.141 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:23:14.143124+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:23:14.141 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:23:14.142 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2060 16 1935 1919 0.0582144 0 - 0.363217 2024-04-16T08:23:15.141 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2061 16 1935 1919 0.0581862 0 - 0.363217 2024-04-16T08:23:16.142 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2062 16 1935 1919 0.0581579 0 - 0.363217 2024-04-16T08:23:17.141 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2063 16 1935 1919 0.0581297 0 - 0.363217 2024-04-16T08:23:18.142 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2064 16 1935 1919 0.0581016 0 - 0.363217 2024-04-16T08:23:19.142 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2065 16 1935 1919 0.0580734 0 - 0.363217 2024-04-16T08:23:20.142 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2066 16 1935 1919 0.0580453 0 - 0.363217 2024-04-16T08:23:21.142 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2067 16 1935 1919 0.0580173 0 - 0.363217 2024-04-16T08:23:22.142 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2068 16 1935 1919 0.0579892 0 - 0.363217 2024-04-16T08:23:23.142 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2069 16 1935 1919 0.0579612 0 - 0.363217 2024-04-16T08:23:24.143 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2070 16 1935 1919 0.0579332 0 - 0.363217 2024-04-16T08:23:25.142 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2071 16 1935 1919 0.0579052 0 - 0.363217 2024-04-16T08:23:26.143 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2072 16 1935 1919 0.0578773 0 - 0.363217 2024-04-16T08:23:27.143 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2073 16 1935 1919 0.0578493 0 - 0.363217 2024-04-16T08:23:28.143 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2074 16 1935 1919 0.0578214 0 - 0.363217 2024-04-16T08:23:29.143 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2075 16 1935 1919 0.0577936 0 - 0.363217 2024-04-16T08:23:30.143 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2076 16 1935 1919 0.0577657 0 - 0.363217 2024-04-16T08:23:31.143 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2077 16 1935 1919 0.0577379 0 - 0.363217 2024-04-16T08:23:32.143 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2078 16 1935 1919 0.0577101 0 - 0.363217 2024-04-16T08:23:33.143 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2079 16 1935 1919 0.0576824 0 - 0.363217 2024-04-16T08:23:34.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:23:34.145573+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:23:34.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:23:34.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2080 16 1935 1919 0.0576547 0 - 0.363217 2024-04-16T08:23:35.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2081 16 1935 1919 0.0576269 0 - 0.363217 2024-04-16T08:23:36.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2082 16 1935 1919 0.0575993 0 - 0.363217 2024-04-16T08:23:37.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2083 16 1935 1919 0.0575716 0 - 0.363217 2024-04-16T08:23:38.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2084 16 1935 1919 0.057544 0 - 0.363217 2024-04-16T08:23:39.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2085 16 1935 1919 0.0575164 0 - 0.363217 2024-04-16T08:23:40.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2086 16 1935 1919 0.0574888 0 - 0.363217 2024-04-16T08:23:41.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2087 16 1935 1919 0.0574613 0 - 0.363217 2024-04-16T08:23:42.144 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2088 16 1935 1919 0.0574338 0 - 0.363217 2024-04-16T08:23:43.145 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2089 16 1935 1919 0.0574063 0 - 0.363217 2024-04-16T08:23:44.145 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2090 16 1935 1919 0.0573788 0 - 0.363217 2024-04-16T08:23:45.145 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2091 16 1935 1919 0.0573514 0 - 0.363217 2024-04-16T08:23:46.145 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2092 16 1935 1919 0.0573239 0 - 0.363217 2024-04-16T08:23:47.145 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2093 16 1935 1919 0.0572966 0 - 0.363217 2024-04-16T08:23:48.145 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2094 16 1935 1919 0.0572692 0 - 0.363217 2024-04-16T08:23:49.146 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2095 16 1935 1919 0.0572419 0 - 0.363217 2024-04-16T08:23:50.146 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2096 16 1935 1919 0.0572145 0 - 0.363217 2024-04-16T08:23:51.146 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2097 16 1935 1919 0.0571873 0 - 0.363217 2024-04-16T08:23:52.146 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2098 16 1935 1919 0.05716 0 - 0.363217 2024-04-16T08:23:53.146 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2099 16 1935 1919 0.0571328 0 - 0.363217 2024-04-16T08:23:54.146 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:23:54.148193+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:23:54.146 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:23:54.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2100 16 1935 1919 0.0571056 0 - 0.363217 2024-04-16T08:23:55.146 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2101 16 1935 1919 0.0570784 0 - 0.363217 2024-04-16T08:23:56.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2102 16 1935 1919 0.0570512 0 - 0.363217 2024-04-16T08:23:57.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2103 16 1935 1919 0.0570241 0 - 0.363217 2024-04-16T08:23:58.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2104 16 1935 1919 0.056997 0 - 0.363217 2024-04-16T08:23:59.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2105 16 1935 1919 0.0569699 0 - 0.363217 2024-04-16T08:24:00.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2106 16 1935 1919 0.0569429 0 - 0.363217 2024-04-16T08:24:01.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2107 16 1935 1919 0.0569158 0 - 0.363217 2024-04-16T08:24:02.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2108 16 1935 1919 0.0568888 0 - 0.363217 2024-04-16T08:24:03.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2109 16 1935 1919 0.0568619 0 - 0.363217 2024-04-16T08:24:04.148 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2110 16 1935 1919 0.0568349 0 - 0.363217 2024-04-16T08:24:05.147 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2111 16 1935 1919 0.056808 0 - 0.363217 2024-04-16T08:24:06.148 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2112 16 1935 1919 0.0567811 0 - 0.363217 2024-04-16T08:24:07.148 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2113 16 1935 1919 0.0567542 0 - 0.363217 2024-04-16T08:24:08.148 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2114 16 1935 1919 0.0567274 0 - 0.363217 2024-04-16T08:24:09.148 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2115 16 1935 1919 0.0567006 0 - 0.363217 2024-04-16T08:24:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:24:09.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:24:10.148 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2116 16 1935 1919 0.0566738 0 - 0.363217 2024-04-16T08:24:11.148 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2117 16 1935 1919 0.056647 0 - 0.363217 2024-04-16T08:24:11.449 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T08:24:11.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:24:11.571+0000 7fba16aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:24:12.148 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2118 16 1935 1919 0.0566203 0 - 0.363217 2024-04-16T08:24:13.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2119 16 1935 1919 0.0565935 0 - 0.363217 2024-04-16T08:24:14.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:24:14.150642+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:24:14.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:24:14.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2120 16 1935 1919 0.0565668 0 - 0.363217 2024-04-16T08:24:15.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2121 16 1935 1919 0.0565402 0 - 0.363217 2024-04-16T08:24:16.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2122 16 1935 1919 0.0565135 0 - 0.363217 2024-04-16T08:24:17.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2123 16 1935 1919 0.0564869 0 - 0.363217 2024-04-16T08:24:18.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2124 16 1935 1919 0.0564603 0 - 0.363217 2024-04-16T08:24:19.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2125 16 1935 1919 0.0564337 0 - 0.363217 2024-04-16T08:24:20.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2126 16 1935 1919 0.0564072 0 - 0.363217 2024-04-16T08:24:21.149 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2127 16 1935 1919 0.0563807 0 - 0.363217 2024-04-16T08:24:22.150 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2128 16 1935 1919 0.0563542 0 - 0.363217 2024-04-16T08:24:23.150 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2129 16 1935 1919 0.0563277 0 - 0.363217 2024-04-16T08:24:24.150 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2130 16 1935 1919 0.0563013 0 - 0.363217 2024-04-16T08:24:25.150 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2131 16 1935 1919 0.0562748 0 - 0.363217 2024-04-16T08:24:26.150 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2132 16 1935 1919 0.0562485 0 - 0.363217 2024-04-16T08:24:27.150 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2133 16 1935 1919 0.0562221 0 - 0.363217 2024-04-16T08:24:28.150 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2134 16 1935 1919 0.0561957 0 - 0.363217 2024-04-16T08:24:29.150 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2135 16 1935 1919 0.0561694 0 - 0.363217 2024-04-16T08:24:30.151 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2136 16 1935 1919 0.0561431 0 - 0.363217 2024-04-16T08:24:31.151 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2137 16 1935 1919 0.0561168 0 - 0.363217 2024-04-16T08:24:32.151 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2138 16 1935 1919 0.0560906 0 - 0.363217 2024-04-16T08:24:33.151 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2139 16 1935 1919 0.0560644 0 - 0.363217 2024-04-16T08:24:34.151 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:24:34.153057+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:24:34.151 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:24:34.151 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2140 16 1935 1919 0.0560382 0 - 0.363217 2024-04-16T08:24:35.151 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2141 16 1935 1919 0.056012 0 - 0.363217 2024-04-16T08:24:36.151 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2142 16 1935 1919 0.0559859 0 - 0.363217 2024-04-16T08:24:37.152 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2143 16 1935 1919 0.0559597 0 - 0.363217 2024-04-16T08:24:38.152 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2144 16 1935 1919 0.0559336 0 - 0.363217 2024-04-16T08:24:39.152 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2145 16 1935 1919 0.0559076 0 - 0.363217 2024-04-16T08:24:40.152 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2146 16 1935 1919 0.0558815 0 - 0.363217 2024-04-16T08:24:41.152 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2147 16 1935 1919 0.0558555 0 - 0.363217 2024-04-16T08:24:42.152 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2148 16 1935 1919 0.0558295 0 - 0.363217 2024-04-16T08:24:43.152 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2149 16 1935 1919 0.0558035 0 - 0.363217 2024-04-16T08:24:44.152 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2150 16 1935 1919 0.0557775 0 - 0.363217 2024-04-16T08:24:45.152 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2151 16 1935 1919 0.0557516 0 - 0.363217 2024-04-16T08:24:46.153 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2152 16 1935 1919 0.0557257 0 - 0.363217 2024-04-16T08:24:47.153 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2153 16 1935 1919 0.0556998 0 - 0.363217 2024-04-16T08:24:48.153 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2154 16 1935 1919 0.055674 0 - 0.363217 2024-04-16T08:24:49.153 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2155 16 1935 1919 0.0556481 0 - 0.363217 2024-04-16T08:24:50.153 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2156 16 1935 1919 0.0556223 0 - 0.363217 2024-04-16T08:24:51.153 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2157 16 1935 1919 0.0555965 0 - 0.363217 2024-04-16T08:24:52.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2158 16 1935 1919 0.0555708 0 - 0.363217 2024-04-16T08:24:53.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2159 16 1935 1919 0.055545 0 - 0.363217 2024-04-16T08:24:54.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:24:54.155659+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:24:54.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:24:54.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2160 16 1935 1919 0.0555193 0 - 0.363217 2024-04-16T08:24:55.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2161 16 1935 1919 0.0554936 0 - 0.363217 2024-04-16T08:24:56.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2162 16 1935 1919 0.0554679 0 - 0.363217 2024-04-16T08:24:57.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2163 16 1935 1919 0.0554423 0 - 0.363217 2024-04-16T08:24:58.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2164 16 1935 1919 0.0554167 0 - 0.363217 2024-04-16T08:24:59.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2165 16 1935 1919 0.0553911 0 - 0.363217 2024-04-16T08:25:00.154 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2166 16 1935 1919 0.0553655 0 - 0.363217 2024-04-16T08:25:01.155 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2167 16 1935 1919 0.05534 0 - 0.363217 2024-04-16T08:25:02.155 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2168 16 1935 1919 0.0553144 0 - 0.363217 2024-04-16T08:25:03.155 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2169 16 1935 1919 0.0552889 0 - 0.363217 2024-04-16T08:25:04.155 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2170 16 1935 1919 0.0552635 0 - 0.363217 2024-04-16T08:25:05.155 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2171 16 1935 1919 0.055238 0 - 0.363217 2024-04-16T08:25:06.155 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2172 16 1935 1919 0.0552126 0 - 0.363217 2024-04-16T08:25:07.155 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2173 16 1935 1919 0.0551872 0 - 0.363217 2024-04-16T08:25:08.156 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2174 16 1935 1919 0.0551618 0 - 0.363217 2024-04-16T08:25:09.156 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2175 16 1935 1919 0.0551364 0 - 0.363217 2024-04-16T08:25:10.156 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2176 16 1935 1919 0.0551111 0 - 0.363217 2024-04-16T08:25:11.156 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2177 16 1935 1919 0.0550858 0 - 0.363217 2024-04-16T08:25:12.156 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2178 16 1935 1919 0.0550605 0 - 0.363217 2024-04-16T08:25:13.156 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2179 16 1935 1919 0.0550352 0 - 0.363217 2024-04-16T08:25:14.156 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:25:14.158307+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:25:14.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:25:14.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2180 16 1935 1919 0.05501 0 - 0.363217 2024-04-16T08:25:15.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2181 16 1935 1919 0.0549847 0 - 0.363217 2024-04-16T08:25:16.156 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2182 16 1935 1919 0.0549595 0 - 0.363217 2024-04-16T08:25:17.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2183 16 1935 1919 0.0549344 0 - 0.363217 2024-04-16T08:25:18.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2184 16 1935 1919 0.0549092 0 - 0.363217 2024-04-16T08:25:19.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2185 16 1935 1919 0.0548841 0 - 0.363217 2024-04-16T08:25:20.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2186 16 1935 1919 0.054859 0 - 0.363217 2024-04-16T08:25:21.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2187 16 1935 1919 0.0548339 0 - 0.363217 2024-04-16T08:25:22.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2188 16 1935 1919 0.0548088 0 - 0.363217 2024-04-16T08:25:23.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2189 16 1935 1919 0.0547838 0 - 0.363217 2024-04-16T08:25:24.157 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2190 16 1935 1919 0.0547588 0 - 0.363217 2024-04-16T08:25:25.158 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2191 16 1935 1919 0.0547338 0 - 0.363217 2024-04-16T08:25:26.158 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2192 16 1935 1919 0.0547088 0 - 0.363217 2024-04-16T08:25:27.158 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2193 16 1935 1919 0.0546839 0 - 0.363217 2024-04-16T08:25:28.158 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2194 16 1935 1919 0.0546589 0 - 0.363217 2024-04-16T08:25:29.158 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2195 16 1935 1919 0.054634 0 - 0.363217 2024-04-16T08:25:30.158 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2196 16 1935 1919 0.0546092 0 - 0.363217 2024-04-16T08:25:31.158 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2197 16 1935 1919 0.0545843 0 - 0.363217 2024-04-16T08:25:32.159 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2198 16 1935 1919 0.0545595 0 - 0.363217 2024-04-16T08:25:33.159 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2199 16 1935 1919 0.0545347 0 - 0.363217 2024-04-16T08:25:34.159 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:25:34.160872+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:25:34.159 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:25:34.159 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2200 16 1935 1919 0.0545099 0 - 0.363217 2024-04-16T08:25:35.159 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2201 16 1935 1919 0.0544851 0 - 0.363217 2024-04-16T08:25:36.159 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2202 16 1935 1919 0.0544604 0 - 0.363217 2024-04-16T08:25:37.159 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2203 16 1935 1919 0.0544356 0 - 0.363217 2024-04-16T08:25:38.160 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2204 16 1935 1919 0.0544109 0 - 0.363217 2024-04-16T08:25:39.160 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2205 16 1935 1919 0.0543863 0 - 0.363217 2024-04-16T08:25:40.160 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2206 16 1935 1919 0.0543616 0 - 0.363217 2024-04-16T08:25:41.160 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2207 16 1935 1919 0.054337 0 - 0.363217 2024-04-16T08:25:42.160 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2208 16 1935 1919 0.0543124 0 - 0.363217 2024-04-16T08:25:43.160 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2209 16 1935 1919 0.0542878 0 - 0.363217 2024-04-16T08:25:44.160 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2210 16 1935 1919 0.0542632 0 - 0.363217 2024-04-16T08:25:45.160 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2211 16 1935 1919 0.0542387 0 - 0.363217 2024-04-16T08:25:46.161 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2212 16 1935 1919 0.0542142 0 - 0.363217 2024-04-16T08:25:47.161 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2213 16 1935 1919 0.0541897 0 - 0.363217 2024-04-16T08:25:48.161 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2214 16 1935 1919 0.0541652 0 - 0.363217 2024-04-16T08:25:49.161 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2215 16 1935 1919 0.0541407 0 - 0.363217 2024-04-16T08:25:50.161 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2216 16 1935 1919 0.0541163 0 - 0.363217 2024-04-16T08:25:51.161 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2217 16 1935 1919 0.0540919 0 - 0.363217 2024-04-16T08:25:52.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2218 16 1935 1919 0.0540675 0 - 0.363217 2024-04-16T08:25:53.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2219 16 1935 1919 0.0540431 0 - 0.363217 2024-04-16T08:25:54.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:25:54.163646+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:25:54.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:25:54.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2220 16 1935 1919 0.0540188 0 - 0.363217 2024-04-16T08:25:55.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2221 16 1935 1919 0.0539945 0 - 0.363217 2024-04-16T08:25:56.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2222 16 1935 1919 0.0539702 0 - 0.363217 2024-04-16T08:25:57.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2223 16 1935 1919 0.0539459 0 - 0.363217 2024-04-16T08:25:58.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2224 16 1935 1919 0.0539216 0 - 0.363217 2024-04-16T08:25:59.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2225 16 1935 1919 0.0538974 0 - 0.363217 2024-04-16T08:26:00.162 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2226 16 1935 1919 0.0538732 0 - 0.363217 2024-04-16T08:26:01.163 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2227 16 1935 1919 0.053849 0 - 0.363217 2024-04-16T08:26:02.163 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2228 16 1935 1919 0.0538248 0 - 0.363217 2024-04-16T08:26:03.163 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2229 16 1935 1919 0.0538007 0 - 0.363217 2024-04-16T08:26:04.163 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2230 16 1935 1919 0.0537765 0 - 0.363217 2024-04-16T08:26:05.163 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2231 16 1935 1919 0.0537524 0 - 0.363217 2024-04-16T08:26:06.163 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2232 16 1935 1919 0.0537284 0 - 0.363217 2024-04-16T08:26:07.163 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2233 16 1935 1919 0.0537043 0 - 0.363217 2024-04-16T08:26:08.164 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2234 16 1935 1919 0.0536803 0 - 0.363217 2024-04-16T08:26:09.164 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2235 16 1935 1919 0.0536562 0 - 0.363217 2024-04-16T08:26:10.164 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2236 16 1935 1919 0.0536322 0 - 0.363217 2024-04-16T08:26:11.164 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2237 16 1935 1919 0.0536083 0 - 0.363217 2024-04-16T08:26:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:26:11.490 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:26:12.164 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2238 16 1935 1919 0.0535843 0 - 0.363217 2024-04-16T08:26:13.164 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2239 16 1935 1919 0.0535604 0 - 0.363217 2024-04-16T08:26:13.491 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:26:13.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:26:13.613+0000 7ff49b39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:26:14.164 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:26:14.166300+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:26:14.165 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:26:14.165 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2240 16 1935 1919 0.0535365 0 - 0.363217 2024-04-16T08:26:15.165 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2241 16 1935 1919 0.0535126 0 - 0.363217 2024-04-16T08:26:16.165 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2242 16 1935 1919 0.0534887 0 - 0.363217 2024-04-16T08:26:17.165 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2243 16 1935 1919 0.0534649 0 - 0.363217 2024-04-16T08:26:18.165 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2244 16 1935 1919 0.053441 0 - 0.363217 2024-04-16T08:26:19.165 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2245 16 1935 1919 0.0534172 0 - 0.363217 2024-04-16T08:26:20.165 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2246 16 1935 1919 0.0533935 0 - 0.363217 2024-04-16T08:26:21.166 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2247 16 1935 1919 0.0533697 0 - 0.363217 2024-04-16T08:26:22.166 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2248 16 1935 1919 0.0533459 0 - 0.363217 2024-04-16T08:26:23.166 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2249 16 1935 1919 0.0533222 0 - 0.363217 2024-04-16T08:26:24.166 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2250 16 1935 1919 0.0532985 0 - 0.363217 2024-04-16T08:26:25.166 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2251 16 1935 1919 0.0532749 0 - 0.363217 2024-04-16T08:26:26.166 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2252 16 1935 1919 0.0532512 0 - 0.363217 2024-04-16T08:26:27.166 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2253 16 1935 1919 0.0532276 0 - 0.363217 2024-04-16T08:26:28.166 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2254 16 1935 1919 0.0532039 0 - 0.363217 2024-04-16T08:26:29.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2255 16 1935 1919 0.0531804 0 - 0.363217 2024-04-16T08:26:30.166 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2256 16 1935 1919 0.0531568 0 - 0.363217 2024-04-16T08:26:31.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2257 16 1935 1919 0.0531332 0 - 0.363217 2024-04-16T08:26:32.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2258 16 1935 1919 0.0531097 0 - 0.363217 2024-04-16T08:26:33.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2259 16 1935 1919 0.0530862 0 - 0.363217 2024-04-16T08:26:34.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:26:34.169067+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:26:34.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:26:34.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2260 16 1935 1919 0.0530627 0 - 0.363217 2024-04-16T08:26:35.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2261 16 1935 1919 0.0530392 0 - 0.363217 2024-04-16T08:26:36.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2262 16 1935 1919 0.0530158 0 - 0.363217 2024-04-16T08:26:37.167 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2263 16 1935 1919 0.0529924 0 - 0.363217 2024-04-16T08:26:38.168 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2264 16 1935 1919 0.0529689 0 - 0.363217 2024-04-16T08:26:39.168 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2265 16 1935 1919 0.0529456 0 - 0.363217 2024-04-16T08:26:40.168 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2266 16 1935 1919 0.0529222 0 - 0.363217 2024-04-16T08:26:41.168 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2267 16 1935 1919 0.0528989 0 - 0.363217 2024-04-16T08:26:42.168 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2268 16 1935 1919 0.0528755 0 - 0.363217 2024-04-16T08:26:43.168 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2269 16 1935 1919 0.0528522 0 - 0.363217 2024-04-16T08:26:44.168 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2270 16 1935 1919 0.0528289 0 - 0.363217 2024-04-16T08:26:45.169 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2271 16 1935 1919 0.0528057 0 - 0.363217 2024-04-16T08:26:46.169 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2272 16 1935 1919 0.0527824 0 - 0.363217 2024-04-16T08:26:47.169 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2273 16 1935 1919 0.0527592 0 - 0.363217 2024-04-16T08:26:48.169 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2274 16 1935 1919 0.052736 0 - 0.363217 2024-04-16T08:26:49.169 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2275 16 1935 1919 0.0527128 0 - 0.363217 2024-04-16T08:26:50.169 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2276 16 1935 1919 0.0526897 0 - 0.363217 2024-04-16T08:26:51.169 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2277 16 1935 1919 0.0526665 0 - 0.363217 2024-04-16T08:26:52.169 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2278 16 1935 1919 0.0526434 0 - 0.363217 2024-04-16T08:26:53.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2279 16 1935 1919 0.0526203 0 - 0.363217 2024-04-16T08:26:54.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:26:54.171628+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:26:54.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:26:54.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2280 16 1935 1919 0.0525972 0 - 0.363217 2024-04-16T08:26:55.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2281 16 1935 1919 0.0525742 0 - 0.363217 2024-04-16T08:26:56.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2282 16 1935 1919 0.0525511 0 - 0.363217 2024-04-16T08:26:57.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2283 16 1935 1919 0.0525281 0 - 0.363217 2024-04-16T08:26:58.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2284 16 1935 1919 0.0525051 0 - 0.363217 2024-04-16T08:26:59.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2285 16 1935 1919 0.0524821 0 - 0.363217 2024-04-16T08:27:00.170 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2286 16 1935 1919 0.0524592 0 - 0.363217 2024-04-16T08:27:01.171 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2287 16 1935 1919 0.0524362 0 - 0.363217 2024-04-16T08:27:02.171 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2288 16 1935 1919 0.0524133 0 - 0.363217 2024-04-16T08:27:03.171 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2289 16 1935 1919 0.0523904 0 - 0.363217 2024-04-16T08:27:04.171 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2290 16 1935 1919 0.0523676 0 - 0.363217 2024-04-16T08:27:05.171 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2291 16 1935 1919 0.0523447 0 - 0.363217 2024-04-16T08:27:06.171 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2292 16 1935 1919 0.0523219 0 - 0.363217 2024-04-16T08:27:07.172 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2293 16 1935 1919 0.052299 0 - 0.363217 2024-04-16T08:27:08.171 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2294 16 1935 1919 0.0522762 0 - 0.363217 2024-04-16T08:27:09.172 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2295 16 1935 1919 0.0522535 0 - 0.363217 2024-04-16T08:27:10.172 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2296 16 1935 1919 0.0522307 0 - 0.363217 2024-04-16T08:27:11.172 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2297 16 1935 1919 0.052208 0 - 0.363217 2024-04-16T08:27:12.172 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2298 16 1935 1919 0.0521852 0 - 0.363217 2024-04-16T08:27:13.172 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2299 16 1935 1919 0.0521625 0 - 0.363217 2024-04-16T08:27:14.172 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:27:14.174222+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:27:14.172 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:27:14.173 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2300 16 1935 1919 0.0521399 0 - 0.363217 2024-04-16T08:27:15.172 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2301 16 1935 1919 0.0521172 0 - 0.363217 2024-04-16T08:27:16.173 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2302 16 1935 1919 0.0520946 0 - 0.363217 2024-04-16T08:27:17.173 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2303 16 1935 1919 0.0520719 0 - 0.363217 2024-04-16T08:27:18.173 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2304 16 1935 1919 0.0520493 0 - 0.363217 2024-04-16T08:27:19.173 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2305 16 1935 1919 0.0520268 0 - 0.363217 2024-04-16T08:27:20.173 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2306 16 1935 1919 0.0520042 0 - 0.363217 2024-04-16T08:27:21.173 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2307 16 1935 1919 0.0519817 0 - 0.363217 2024-04-16T08:27:22.173 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2308 16 1935 1919 0.0519591 0 - 0.363217 2024-04-16T08:27:23.173 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2309 16 1935 1919 0.0519366 0 - 0.363217 2024-04-16T08:27:24.174 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2310 16 1935 1919 0.0519142 0 - 0.363217 2024-04-16T08:27:25.174 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2311 16 1935 1919 0.0518917 0 - 0.363217 2024-04-16T08:27:26.174 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2312 16 1935 1919 0.0518692 0 - 0.363217 2024-04-16T08:27:27.174 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2313 16 1935 1919 0.0518468 0 - 0.363217 2024-04-16T08:27:28.174 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2314 16 1935 1919 0.0518244 0 - 0.363217 2024-04-16T08:27:29.174 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2315 16 1935 1919 0.051802 0 - 0.363217 2024-04-16T08:27:30.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2316 16 1935 1919 0.0517797 0 - 0.363217 2024-04-16T08:27:31.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2317 16 1935 1919 0.0517573 0 - 0.363217 2024-04-16T08:27:32.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2318 16 1935 1919 0.051735 0 - 0.363217 2024-04-16T08:27:33.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2319 16 1935 1919 0.0517127 0 - 0.363217 2024-04-16T08:27:34.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:27:34.176966+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:27:34.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:27:34.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2320 16 1935 1919 0.0516904 0 - 0.363217 2024-04-16T08:27:35.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2321 16 1935 1919 0.0516681 0 - 0.363217 2024-04-16T08:27:36.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2322 16 1935 1919 0.0516459 0 - 0.363217 2024-04-16T08:27:37.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2323 16 1935 1919 0.0516236 0 - 0.363217 2024-04-16T08:27:38.175 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2324 16 1935 1919 0.0516014 0 - 0.363217 2024-04-16T08:27:39.176 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2325 16 1935 1919 0.0515792 0 - 0.363217 2024-04-16T08:27:40.176 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2326 16 1935 1919 0.051557 0 - 0.363217 2024-04-16T08:27:41.176 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2327 16 1935 1919 0.0515349 0 - 0.363217 2024-04-16T08:27:42.176 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2328 16 1935 1919 0.0515128 0 - 0.363217 2024-04-16T08:27:43.176 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2329 16 1935 1919 0.0514906 0 - 0.363217 2024-04-16T08:27:44.176 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2330 16 1935 1919 0.0514685 0 - 0.363217 2024-04-16T08:27:45.177 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2331 16 1935 1919 0.0514465 0 - 0.363217 2024-04-16T08:27:46.177 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2332 16 1935 1919 0.0514244 0 - 0.363217 2024-04-16T08:27:47.177 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2333 16 1935 1919 0.0514024 0 - 0.363217 2024-04-16T08:27:48.177 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2334 16 1935 1919 0.0513803 0 - 0.363217 2024-04-16T08:27:49.177 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2335 16 1935 1919 0.0513583 0 - 0.363217 2024-04-16T08:27:50.177 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2336 16 1935 1919 0.0513363 0 - 0.363217 2024-04-16T08:27:51.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2337 16 1935 1919 0.0513144 0 - 0.363217 2024-04-16T08:27:52.177 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2338 16 1935 1919 0.0512924 0 - 0.363217 2024-04-16T08:27:53.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2339 16 1935 1919 0.0512705 0 - 0.363217 2024-04-16T08:27:54.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:27:54.179632+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:27:54.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:27:54.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2340 16 1935 1919 0.0512486 0 - 0.363217 2024-04-16T08:27:55.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2341 16 1935 1919 0.0512267 0 - 0.363217 2024-04-16T08:27:56.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2342 16 1935 1919 0.0512048 0 - 0.363217 2024-04-16T08:27:57.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2343 16 1935 1919 0.051183 0 - 0.363217 2024-04-16T08:27:58.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2344 16 1935 1919 0.0511611 0 - 0.363217 2024-04-16T08:27:59.178 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2345 16 1935 1919 0.0511393 0 - 0.363217 2024-04-16T08:28:00.179 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2346 16 1935 1919 0.0511175 0 - 0.363217 2024-04-16T08:28:01.179 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2347 16 1935 1919 0.0510957 0 - 0.363217 2024-04-16T08:28:02.179 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2348 16 1935 1919 0.051074 0 - 0.363217 2024-04-16T08:28:03.179 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2349 16 1935 1919 0.0510522 0 - 0.363217 2024-04-16T08:28:04.179 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2350 16 1935 1919 0.0510305 0 - 0.363217 2024-04-16T08:28:05.179 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2351 16 1935 1919 0.0510088 0 - 0.363217 2024-04-16T08:28:06.179 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2352 16 1935 1919 0.0509871 0 - 0.363217 2024-04-16T08:28:07.180 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2353 16 1935 1919 0.0509654 0 - 0.363217 2024-04-16T08:28:08.180 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2354 16 1935 1919 0.0509438 0 - 0.363217 2024-04-16T08:28:09.180 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2355 16 1935 1919 0.0509222 0 - 0.363217 2024-04-16T08:28:10.180 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2356 16 1935 1919 0.0509005 0 - 0.363217 2024-04-16T08:28:11.180 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2357 16 1935 1919 0.050879 0 - 0.363217 2024-04-16T08:28:12.180 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2358 16 1935 1919 0.0508574 0 - 0.363217 2024-04-16T08:28:13.180 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2359 16 1935 1919 0.0508358 0 - 0.363217 2024-04-16T08:28:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:28:13.532 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:28:14.180 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:28:14.182387+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:28:14.181 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:28:14.181 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2360 16 1935 1919 0.0508143 0 - 0.363217 2024-04-16T08:28:15.181 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2361 16 1935 1919 0.0507928 0 - 0.363217 2024-04-16T08:28:15.534 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T08:28:15.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:28:15.655+0000 7f899026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:28:16.181 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2362 16 1935 1919 0.0507712 0 - 0.363217 2024-04-16T08:28:17.181 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2363 16 1935 1919 0.0507498 0 - 0.363217 2024-04-16T08:28:18.181 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2364 16 1935 1919 0.0507283 0 - 0.363217 2024-04-16T08:28:19.181 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2365 16 1935 1919 0.0507068 0 - 0.363217 2024-04-16T08:28:20.181 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2366 16 1935 1919 0.0506854 0 - 0.363217 2024-04-16T08:28:21.182 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2367 16 1935 1919 0.050664 0 - 0.363217 2024-04-16T08:28:22.181 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2368 16 1935 1919 0.0506426 0 - 0.363217 2024-04-16T08:28:23.182 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2369 16 1935 1919 0.0506212 0 - 0.363217 2024-04-16T08:28:24.182 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2370 16 1935 1919 0.0505999 0 - 0.363217 2024-04-16T08:28:25.182 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2371 16 1935 1919 0.0505785 0 - 0.363217 2024-04-16T08:28:26.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2372 16 1935 1919 0.0505572 0 - 0.363217 2024-04-16T08:28:27.182 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2373 16 1935 1919 0.0505359 0 - 0.363217 2024-04-16T08:28:28.182 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2374 16 1935 1919 0.0505146 0 - 0.363217 2024-04-16T08:28:29.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2375 16 1935 1919 0.0504933 0 - 0.363217 2024-04-16T08:28:30.182 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2376 16 1935 1919 0.0504721 0 - 0.363217 2024-04-16T08:28:31.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2377 16 1935 1919 0.0504509 0 - 0.363217 2024-04-16T08:28:32.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2378 16 1935 1919 0.0504296 0 - 0.363217 2024-04-16T08:28:33.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2379 16 1935 1919 0.0504084 0 - 0.363217 2024-04-16T08:28:34.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:28:34.184965+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:28:34.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:28:34.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2380 16 1935 1919 0.0503873 0 - 0.363217 2024-04-16T08:28:35.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2381 16 1935 1919 0.0503661 0 - 0.363217 2024-04-16T08:28:36.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2382 16 1935 1919 0.050345 0 - 0.363217 2024-04-16T08:28:37.183 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2383 16 1935 1919 0.0503238 0 - 0.363217 2024-04-16T08:28:38.184 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2384 16 1935 1919 0.0503027 0 - 0.363217 2024-04-16T08:28:39.184 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2385 16 1935 1919 0.0502816 0 - 0.363217 2024-04-16T08:28:40.184 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2386 16 1935 1919 0.0502606 0 - 0.363217 2024-04-16T08:28:41.184 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2387 16 1935 1919 0.0502395 0 - 0.363217 2024-04-16T08:28:42.184 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2388 16 1935 1919 0.0502185 0 - 0.363217 2024-04-16T08:28:43.184 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2389 16 1935 1919 0.0501974 0 - 0.363217 2024-04-16T08:28:44.185 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2390 16 1935 1919 0.0501764 0 - 0.363217 2024-04-16T08:28:45.185 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2391 16 1935 1919 0.0501555 0 - 0.363217 2024-04-16T08:28:46.185 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2392 16 1935 1919 0.0501345 0 - 0.363217 2024-04-16T08:28:47.185 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2393 16 1935 1919 0.0501135 0 - 0.363217 2024-04-16T08:28:48.185 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2394 16 1935 1919 0.0500926 0 - 0.363217 2024-04-16T08:28:49.185 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2395 16 1935 1919 0.0500717 0 - 0.363217 2024-04-16T08:28:50.185 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2396 16 1935 1919 0.0500508 0 - 0.363217 2024-04-16T08:28:51.185 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2397 16 1935 1919 0.0500299 0 - 0.363217 2024-04-16T08:28:52.185 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2398 16 1935 1919 0.050009 0 - 0.363217 2024-04-16T08:28:53.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2399 16 1935 1919 0.0499882 0 - 0.363217 2024-04-16T08:28:54.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:28:54.187537+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:28:54.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:28:54.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2400 16 1935 1919 0.0499674 0 - 0.363217 2024-04-16T08:28:55.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2401 16 1935 1919 0.0499466 0 - 0.363217 2024-04-16T08:28:56.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2402 16 1935 1919 0.0499258 0 - 0.363217 2024-04-16T08:28:57.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2403 16 1935 1919 0.049905 0 - 0.363217 2024-04-16T08:28:58.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2404 16 1935 1919 0.0498842 0 - 0.363217 2024-04-16T08:28:59.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2405 16 1935 1919 0.0498635 0 - 0.363217 2024-04-16T08:29:00.186 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2406 16 1935 1919 0.0498428 0 - 0.363217 2024-04-16T08:29:01.187 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2407 16 1935 1919 0.0498221 0 - 0.363217 2024-04-16T08:29:02.187 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2408 16 1935 1919 0.0498014 0 - 0.363217 2024-04-16T08:29:03.187 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2409 16 1935 1919 0.0497807 0 - 0.363217 2024-04-16T08:29:04.187 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2410 16 1935 1919 0.04976 0 - 0.363217 2024-04-16T08:29:05.187 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2411 16 1935 1919 0.0497394 0 - 0.363217 2024-04-16T08:29:06.187 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2412 16 1935 1919 0.0497188 0 - 0.363217 2024-04-16T08:29:07.188 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2413 16 1935 1919 0.0496982 0 - 0.363217 2024-04-16T08:29:08.188 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2414 16 1935 1919 0.0496776 0 - 0.363217 2024-04-16T08:29:09.188 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2415 16 1935 1919 0.049657 0 - 0.363217 2024-04-16T08:29:10.188 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2416 16 1935 1919 0.0496365 0 - 0.363217 2024-04-16T08:29:11.188 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2417 16 1935 1919 0.0496159 0 - 0.363217 2024-04-16T08:29:12.188 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2418 16 1935 1919 0.0495954 0 - 0.363217 2024-04-16T08:29:13.188 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2419 16 1935 1919 0.0495749 0 - 0.363217 2024-04-16T08:29:14.188 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:29:14.190229+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:29:14.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:29:14.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2420 16 1935 1919 0.0495544 0 - 0.363217 2024-04-16T08:29:15.188 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2421 16 1935 1919 0.0495339 0 - 0.363217 2024-04-16T08:29:16.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2422 16 1935 1919 0.0495135 0 - 0.363217 2024-04-16T08:29:17.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2423 16 1935 1919 0.0494931 0 - 0.363217 2024-04-16T08:29:18.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2424 16 1935 1919 0.0494726 0 - 0.363217 2024-04-16T08:29:19.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2425 16 1935 1919 0.0494522 0 - 0.363217 2024-04-16T08:29:20.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2426 16 1935 1919 0.0494319 0 - 0.363217 2024-04-16T08:29:21.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2427 16 1935 1919 0.0494115 0 - 0.363217 2024-04-16T08:29:22.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2428 16 1935 1919 0.0493911 0 - 0.363217 2024-04-16T08:29:23.189 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2429 16 1935 1919 0.0493708 0 - 0.363217 2024-04-16T08:29:24.190 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2430 16 1935 1919 0.0493505 0 - 0.363217 2024-04-16T08:29:25.190 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2431 16 1935 1919 0.0493302 0 - 0.363217 2024-04-16T08:29:26.190 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2432 16 1935 1919 0.0493099 0 - 0.363217 2024-04-16T08:29:27.190 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2433 16 1935 1919 0.0492896 0 - 0.363217 2024-04-16T08:29:28.190 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2434 16 1935 1919 0.0492694 0 - 0.363217 2024-04-16T08:29:29.190 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2435 16 1935 1919 0.0492492 0 - 0.363217 2024-04-16T08:29:30.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2436 16 1935 1919 0.0492289 0 - 0.363217 2024-04-16T08:29:31.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2437 16 1935 1919 0.0492087 0 - 0.363217 2024-04-16T08:29:32.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2438 16 1935 1919 0.0491886 0 - 0.363217 2024-04-16T08:29:33.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2439 16 1935 1919 0.0491684 0 - 0.363217 2024-04-16T08:29:34.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:29:34.192957+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:29:34.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:29:34.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2440 16 1935 1919 0.0491482 0 - 0.363217 2024-04-16T08:29:35.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2441 16 1935 1919 0.0491281 0 - 0.363217 2024-04-16T08:29:36.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2442 16 1935 1919 0.049108 0 - 0.363217 2024-04-16T08:29:37.191 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2443 16 1935 1919 0.0490879 0 - 0.363217 2024-04-16T08:29:38.192 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2444 16 1935 1919 0.0490678 0 - 0.363217 2024-04-16T08:29:39.192 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2445 16 1935 1919 0.0490477 0 - 0.363217 2024-04-16T08:29:40.192 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2446 16 1935 1919 0.0490277 0 - 0.363217 2024-04-16T08:29:41.192 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2447 16 1935 1919 0.0490076 0 - 0.363217 2024-04-16T08:29:42.192 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2448 16 1935 1919 0.0489876 0 - 0.363217 2024-04-16T08:29:43.192 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2449 16 1935 1919 0.0489676 0 - 0.363217 2024-04-16T08:29:44.193 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2450 16 1935 1919 0.0489476 0 - 0.363217 2024-04-16T08:29:45.193 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2451 16 1935 1919 0.0489277 0 - 0.363217 2024-04-16T08:29:46.193 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2452 16 1935 1919 0.0489077 0 - 0.363217 2024-04-16T08:29:47.193 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2453 16 1935 1919 0.0488878 0 - 0.363217 2024-04-16T08:29:48.193 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2454 16 1935 1919 0.0488678 0 - 0.363217 2024-04-16T08:29:49.193 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2455 16 1935 1919 0.0488479 0 - 0.363217 2024-04-16T08:29:50.193 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2456 16 1935 1919 0.048828 0 - 0.363217 2024-04-16T08:29:51.193 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2457 16 1935 1919 0.0488082 0 - 0.363217 2024-04-16T08:29:52.194 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2458 16 1935 1919 0.0487883 0 - 0.363217 2024-04-16T08:29:53.194 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2459 16 1935 1919 0.0487685 0 - 0.363217 2024-04-16T08:29:54.194 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:29:54.195736+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:29:54.194 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:29:54.194 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2460 16 1935 1919 0.0487486 0 - 0.363217 2024-04-16T08:29:55.194 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2461 16 1935 1919 0.0487288 0 - 0.363217 2024-04-16T08:29:56.194 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2462 16 1935 1919 0.048709 0 - 0.363217 2024-04-16T08:29:57.194 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2463 16 1935 1919 0.0486893 0 - 0.363217 2024-04-16T08:29:58.194 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2464 16 1935 1919 0.0486695 0 - 0.363217 2024-04-16T08:29:59.195 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2465 16 1935 1919 0.0486498 0 - 0.363217 2024-04-16T08:30:00.195 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2466 16 1935 1919 0.04863 0 - 0.363217 2024-04-16T08:30:01.195 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2467 16 1935 1919 0.0486103 0 - 0.363217 2024-04-16T08:30:02.195 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2468 16 1935 1919 0.0485906 0 - 0.363217 2024-04-16T08:30:03.195 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2469 16 1935 1919 0.0485709 0 - 0.363217 2024-04-16T08:30:04.195 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2470 16 1935 1919 0.0485513 0 - 0.363217 2024-04-16T08:30:05.196 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2471 16 1935 1919 0.0485316 0 - 0.363217 2024-04-16T08:30:06.196 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2472 16 1935 1919 0.048512 0 - 0.363217 2024-04-16T08:30:07.196 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2473 16 1935 1919 0.0484924 0 - 0.363217 2024-04-16T08:30:08.196 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2474 16 1935 1919 0.0484728 0 - 0.363217 2024-04-16T08:30:09.196 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2475 16 1935 1919 0.0484532 0 - 0.363217 2024-04-16T08:30:10.196 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2476 16 1935 1919 0.0484336 0 - 0.363217 2024-04-16T08:30:11.196 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2477 16 1935 1919 0.0484141 0 - 0.363217 2024-04-16T08:30:12.196 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2478 16 1935 1919 0.0483945 0 - 0.363217 2024-04-16T08:30:13.196 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2479 16 1935 1919 0.048375 0 - 0.363217 2024-04-16T08:30:14.197 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:30:14.198507+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:30:14.197 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:30:14.197 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2480 16 1935 1919 0.0483555 0 - 0.363217 2024-04-16T08:30:15.197 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2481 16 1935 1919 0.048336 0 - 0.363217 2024-04-16T08:30:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:30:15.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:30:16.197 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2482 16 1935 1919 0.0483165 0 - 0.363217 2024-04-16T08:30:17.197 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2483 16 1935 1919 0.0482971 0 - 0.363217 2024-04-16T08:30:17.575 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:30:17.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:30:17.697+0000 7ff0f879d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:30:18.197 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2484 16 1935 1919 0.0482776 0 - 0.363217 2024-04-16T08:30:19.197 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2485 16 1935 1919 0.0482582 0 - 0.363217 2024-04-16T08:30:20.197 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2486 16 1935 1919 0.0482388 0 - 0.363217 2024-04-16T08:30:21.198 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2487 16 1935 1919 0.0482194 0 - 0.363217 2024-04-16T08:30:22.198 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2488 16 1935 1919 0.0482 0 - 0.363217 2024-04-16T08:30:23.198 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2489 16 1935 1919 0.0481807 0 - 0.363217 2024-04-16T08:30:24.198 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2490 16 1935 1919 0.0481613 0 - 0.363217 2024-04-16T08:30:25.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2491 16 1935 1919 0.048142 0 - 0.363217 2024-04-16T08:30:26.198 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2492 16 1935 1919 0.0481227 0 - 0.363217 2024-04-16T08:30:27.198 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2493 16 1935 1919 0.0481034 0 - 0.363217 2024-04-16T08:30:28.198 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2494 16 1935 1919 0.0480841 0 - 0.363217 2024-04-16T08:30:29.198 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2495 16 1935 1919 0.0480648 0 - 0.363217 2024-04-16T08:30:30.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2496 16 1935 1919 0.0480455 0 - 0.363217 2024-04-16T08:30:31.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2497 16 1935 1919 0.0480263 0 - 0.363217 2024-04-16T08:30:32.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2498 16 1935 1919 0.0480071 0 - 0.363217 2024-04-16T08:30:33.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2499 16 1935 1919 0.0479879 0 - 0.363217 2024-04-16T08:30:34.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:30:34.200987+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:30:34.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:30:34.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2500 16 1935 1919 0.0479687 0 - 0.363217 2024-04-16T08:30:35.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2501 16 1935 1919 0.0479495 0 - 0.363217 2024-04-16T08:30:36.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2502 16 1935 1919 0.0479303 0 - 0.363217 2024-04-16T08:30:37.199 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2503 16 1935 1919 0.0479112 0 - 0.363217 2024-04-16T08:30:38.200 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2504 16 1935 1919 0.047892 0 - 0.363217 2024-04-16T08:30:39.200 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2505 16 1935 1919 0.0478729 0 - 0.363217 2024-04-16T08:30:40.200 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2506 16 1935 1919 0.0478538 0 - 0.363217 2024-04-16T08:30:41.200 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2507 16 1935 1919 0.0478347 0 - 0.363217 2024-04-16T08:30:42.200 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2508 16 1935 1919 0.0478157 0 - 0.363217 2024-04-16T08:30:43.200 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2509 16 1935 1919 0.0477966 0 - 0.363217 2024-04-16T08:30:44.200 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2510 16 1935 1919 0.0477776 0 - 0.363217 2024-04-16T08:30:45.200 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2511 16 1935 1919 0.0477585 0 - 0.363217 2024-04-16T08:30:46.201 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2512 16 1935 1919 0.0477395 0 - 0.363217 2024-04-16T08:30:47.201 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2513 16 1935 1919 0.0477205 0 - 0.363217 2024-04-16T08:30:48.201 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2514 16 1935 1919 0.0477015 0 - 0.363217 2024-04-16T08:30:49.201 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2515 16 1935 1919 0.0476826 0 - 0.363217 2024-04-16T08:30:50.201 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2516 16 1935 1919 0.0476636 0 - 0.363217 2024-04-16T08:30:51.201 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2517 16 1935 1919 0.0476447 0 - 0.363217 2024-04-16T08:30:52.201 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2518 16 1935 1919 0.0476258 0 - 0.363217 2024-04-16T08:30:53.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2519 16 1935 1919 0.0476069 0 - 0.363217 2024-04-16T08:30:54.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:30:54.203586+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:30:54.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:30:54.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2520 16 1935 1919 0.047588 0 - 0.363217 2024-04-16T08:30:55.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2521 16 1935 1919 0.0475691 0 - 0.363217 2024-04-16T08:30:56.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2522 16 1935 1919 0.0475502 0 - 0.363217 2024-04-16T08:30:57.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2523 16 1935 1919 0.0475314 0 - 0.363217 2024-04-16T08:30:58.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2524 16 1935 1919 0.0475126 0 - 0.363217 2024-04-16T08:30:59.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2525 16 1935 1919 0.0474937 0 - 0.363217 2024-04-16T08:31:00.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2526 16 1935 1919 0.0474749 0 - 0.363217 2024-04-16T08:31:01.202 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2527 16 1935 1919 0.0474561 0 - 0.363217 2024-04-16T08:31:02.203 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2528 16 1935 1919 0.0474374 0 - 0.363217 2024-04-16T08:31:03.203 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2529 16 1935 1919 0.0474186 0 - 0.363217 2024-04-16T08:31:04.203 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2530 16 1935 1919 0.0473999 0 - 0.363217 2024-04-16T08:31:05.203 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2531 16 1935 1919 0.0473811 0 - 0.363217 2024-04-16T08:31:06.203 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2532 16 1935 1919 0.0473624 0 - 0.363217 2024-04-16T08:31:07.203 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2533 16 1935 1919 0.0473437 0 - 0.363217 2024-04-16T08:31:08.203 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2534 16 1935 1919 0.0473251 0 - 0.363217 2024-04-16T08:31:09.203 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2535 16 1935 1919 0.0473064 0 - 0.363217 2024-04-16T08:31:10.204 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2536 16 1935 1919 0.0472877 0 - 0.363217 2024-04-16T08:31:11.204 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2537 16 1935 1919 0.0472691 0 - 0.363217 2024-04-16T08:31:12.204 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2538 16 1935 1919 0.0472505 0 - 0.363217 2024-04-16T08:31:13.204 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2539 16 1935 1919 0.0472319 0 - 0.363217 2024-04-16T08:31:14.204 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:31:14.206077+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:31:14.204 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:31:14.204 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2540 16 1935 1919 0.0472133 0 - 0.363217 2024-04-16T08:31:15.204 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2541 16 1935 1919 0.0471947 0 - 0.363217 2024-04-16T08:31:16.204 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2542 16 1935 1919 0.0471761 0 - 0.363217 2024-04-16T08:31:17.205 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2543 16 1935 1919 0.0471576 0 - 0.363217 2024-04-16T08:31:18.205 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2544 16 1935 1919 0.047139 0 - 0.363217 2024-04-16T08:31:19.205 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2545 16 1935 1919 0.0471205 0 - 0.363217 2024-04-16T08:31:20.205 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2546 16 1935 1919 0.047102 0 - 0.363217 2024-04-16T08:31:21.205 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2547 16 1935 1919 0.0470835 0 - 0.363217 2024-04-16T08:31:22.205 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2548 16 1935 1919 0.047065 0 - 0.363217 2024-04-16T08:31:23.205 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2549 16 1935 1919 0.0470466 0 - 0.363217 2024-04-16T08:31:24.205 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2550 16 1935 1919 0.0470281 0 - 0.363217 2024-04-16T08:31:25.206 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2551 16 1935 1919 0.0470097 0 - 0.363217 2024-04-16T08:31:26.205 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2552 16 1935 1919 0.0469913 0 - 0.363217 2024-04-16T08:31:27.206 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2553 16 1935 1919 0.0469729 0 - 0.363217 2024-04-16T08:31:28.206 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2554 16 1935 1919 0.0469545 0 - 0.363217 2024-04-16T08:31:29.206 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2555 16 1935 1919 0.0469361 0 - 0.363217 2024-04-16T08:31:30.206 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2556 16 1935 1919 0.0469177 0 - 0.363217 2024-04-16T08:31:31.206 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2557 16 1935 1919 0.0468994 0 - 0.363217 2024-04-16T08:31:32.206 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2558 16 1935 1919 0.046881 0 - 0.363217 2024-04-16T08:31:33.206 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2559 16 1935 1919 0.0468627 0 - 0.363217 2024-04-16T08:31:34.206 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:31:34.208488+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:31:34.207 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:31:34.207 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2560 16 1935 1919 0.0468444 0 - 0.363217 2024-04-16T08:31:35.207 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2561 16 1935 1919 0.0468261 0 - 0.363217 2024-04-16T08:31:36.207 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2562 16 1935 1919 0.0468078 0 - 0.363217 2024-04-16T08:31:37.207 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2563 16 1935 1919 0.0467896 0 - 0.363217 2024-04-16T08:31:38.207 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2564 16 1935 1919 0.0467713 0 - 0.363217 2024-04-16T08:31:39.207 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2565 16 1935 1919 0.0467531 0 - 0.363217 2024-04-16T08:31:40.207 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2566 16 1935 1919 0.0467349 0 - 0.363217 2024-04-16T08:31:41.207 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2567 16 1935 1919 0.0467167 0 - 0.363217 2024-04-16T08:31:42.208 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2568 16 1935 1919 0.0466985 0 - 0.363217 2024-04-16T08:31:43.208 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2569 16 1935 1919 0.0466803 0 - 0.363217 2024-04-16T08:31:44.208 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2570 16 1935 1919 0.0466621 0 - 0.363217 2024-04-16T08:31:45.208 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2571 16 1935 1919 0.046644 0 - 0.363217 2024-04-16T08:31:46.208 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2572 16 1935 1919 0.0466259 0 - 0.363217 2024-04-16T08:31:47.208 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2573 16 1935 1919 0.0466077 0 - 0.363217 2024-04-16T08:31:48.208 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2574 16 1935 1919 0.0465896 0 - 0.363217 2024-04-16T08:31:49.209 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2575 16 1935 1919 0.0465715 0 - 0.363217 2024-04-16T08:31:50.209 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2576 16 1935 1919 0.0465535 0 - 0.363217 2024-04-16T08:31:51.209 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2577 16 1935 1919 0.0465354 0 - 0.363217 2024-04-16T08:31:52.209 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2578 16 1935 1919 0.0465173 0 - 0.363217 2024-04-16T08:31:53.209 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2579 16 1935 1919 0.0464993 0 - 0.363217 2024-04-16T08:31:54.209 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:31:54.211254+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:31:54.209 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:31:54.210 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2580 16 1935 1919 0.0464813 0 - 0.363217 2024-04-16T08:31:55.209 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2581 16 1935 1919 0.0464633 0 - 0.363217 2024-04-16T08:31:56.210 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2582 16 1935 1919 0.0464453 0 - 0.363217 2024-04-16T08:31:57.210 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2583 16 1935 1919 0.0464273 0 - 0.363217 2024-04-16T08:31:58.210 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2584 16 1935 1919 0.0464093 0 - 0.363217 2024-04-16T08:31:59.210 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2585 16 1935 1919 0.0463914 0 - 0.363217 2024-04-16T08:32:00.210 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2586 16 1935 1919 0.0463734 0 - 0.363217 2024-04-16T08:32:01.210 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2587 16 1935 1919 0.0463555 0 - 0.363217 2024-04-16T08:32:02.210 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2588 16 1935 1919 0.0463376 0 - 0.363217 2024-04-16T08:32:03.210 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2589 16 1935 1919 0.0463197 0 - 0.363217 2024-04-16T08:32:04.211 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2590 16 1935 1919 0.0463018 0 - 0.363217 2024-04-16T08:32:05.211 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2591 16 1935 1919 0.0462839 0 - 0.363217 2024-04-16T08:32:06.211 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2592 16 1935 1919 0.0462661 0 - 0.363217 2024-04-16T08:32:07.211 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2593 16 1935 1919 0.0462482 0 - 0.363217 2024-04-16T08:32:08.211 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2594 16 1935 1919 0.0462304 0 - 0.363217 2024-04-16T08:32:09.211 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2595 16 1935 1919 0.0462126 0 - 0.363217 2024-04-16T08:32:10.211 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2596 16 1935 1919 0.0461948 0 - 0.363217 2024-04-16T08:32:11.212 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2597 16 1935 1919 0.046177 0 - 0.363217 2024-04-16T08:32:12.213 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2598 16 1935 1919 0.0461592 0 - 0.363217 2024-04-16T08:32:13.214 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2599 16 1935 1919 0.0461415 0 - 0.363217 2024-04-16T08:32:14.212 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:32:14.213866+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:32:14.212 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:32:14.212 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2600 16 1935 1919 0.0461237 0 - 0.363217 2024-04-16T08:32:15.212 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2601 16 1935 1919 0.046106 0 - 0.363217 2024-04-16T08:32:16.212 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2602 16 1935 1919 0.0460883 0 - 0.363217 2024-04-16T08:32:17.212 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2603 16 1935 1919 0.0460706 0 - 0.363217 2024-04-16T08:32:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:32:17.616 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:32:18.213 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2604 16 1935 1919 0.0460529 0 - 0.363217 2024-04-16T08:32:19.213 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2605 16 1935 1919 0.0460352 0 - 0.363217 2024-04-16T08:32:19.618 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T08:32:19.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:32:19.739+0000 7f5ed9aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:32:20.213 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2606 16 1935 1919 0.0460175 0 - 0.363217 2024-04-16T08:32:21.213 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2607 16 1935 1919 0.0459999 0 - 0.363217 2024-04-16T08:32:22.213 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2608 16 1935 1919 0.0459822 0 - 0.363217 2024-04-16T08:32:23.213 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2609 16 1935 1919 0.0459646 0 - 0.363217 2024-04-16T08:32:24.213 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2610 16 1935 1919 0.045947 0 - 0.363217 2024-04-16T08:32:25.213 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2611 16 1935 1919 0.0459294 0 - 0.363217 2024-04-16T08:32:26.214 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2612 16 1935 1919 0.0459118 0 - 0.363217 2024-04-16T08:32:27.214 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2613 16 1935 1919 0.0458943 0 - 0.363217 2024-04-16T08:32:28.214 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2614 16 1935 1919 0.0458767 0 - 0.363217 2024-04-16T08:32:29.214 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2615 16 1935 1919 0.0458592 0 - 0.363217 2024-04-16T08:32:30.214 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2616 16 1935 1919 0.0458416 0 - 0.363217 2024-04-16T08:32:31.214 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2617 16 1935 1919 0.0458241 0 - 0.363217 2024-04-16T08:32:32.214 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2618 16 1935 1919 0.0458066 0 - 0.363217 2024-04-16T08:32:33.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2619 16 1935 1919 0.0457891 0 - 0.363217 2024-04-16T08:32:34.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:32:34.216564+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:32:34.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:32:34.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2620 16 1935 1919 0.0457716 0 - 0.363217 2024-04-16T08:32:35.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2621 16 1935 1919 0.0457542 0 - 0.363217 2024-04-16T08:32:36.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2622 16 1935 1919 0.0457367 0 - 0.363217 2024-04-16T08:32:37.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2623 16 1935 1919 0.0457193 0 - 0.363217 2024-04-16T08:32:38.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2624 16 1935 1919 0.0457019 0 - 0.363217 2024-04-16T08:32:39.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2625 16 1935 1919 0.0456845 0 - 0.363217 2024-04-16T08:32:40.215 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2626 16 1935 1919 0.0456671 0 - 0.363217 2024-04-16T08:32:41.216 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2627 16 1935 1919 0.0456497 0 - 0.363217 2024-04-16T08:32:42.216 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2628 16 1935 1919 0.0456323 0 - 0.363217 2024-04-16T08:32:43.216 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2629 16 1935 1919 0.0456149 0 - 0.363217 2024-04-16T08:32:44.216 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2630 16 1935 1919 0.0455976 0 - 0.363217 2024-04-16T08:32:45.216 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2631 16 1935 1919 0.0455803 0 - 0.363217 2024-04-16T08:32:46.216 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2632 16 1935 1919 0.045563 0 - 0.363217 2024-04-16T08:32:47.216 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2633 16 1935 1919 0.0455456 0 - 0.363217 2024-04-16T08:32:48.216 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2634 16 1935 1919 0.0455284 0 - 0.363217 2024-04-16T08:32:49.217 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2635 16 1935 1919 0.0455111 0 - 0.363217 2024-04-16T08:32:50.217 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2636 16 1935 1919 0.0454938 0 - 0.363217 2024-04-16T08:32:51.217 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2637 16 1935 1919 0.0454766 0 - 0.363217 2024-04-16T08:32:52.217 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2638 16 1935 1919 0.0454593 0 - 0.363217 2024-04-16T08:32:53.217 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2639 16 1935 1919 0.0454421 0 - 0.363217 2024-04-16T08:32:54.217 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:32:54.219303+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:32:54.217 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:32:54.218 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2640 16 1935 1919 0.0454249 0 - 0.363217 2024-04-16T08:32:55.217 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2641 16 1935 1919 0.0454077 0 - 0.363217 2024-04-16T08:32:56.218 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2642 16 1935 1919 0.0453905 0 - 0.363217 2024-04-16T08:32:57.218 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2643 16 1935 1919 0.0453733 0 - 0.363217 2024-04-16T08:32:58.218 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2644 16 1935 1919 0.0453562 0 - 0.363217 2024-04-16T08:32:59.218 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2645 16 1935 1919 0.045339 0 - 0.363217 2024-04-16T08:33:00.218 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2646 16 1935 1919 0.0453219 0 - 0.363217 2024-04-16T08:33:01.218 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2647 16 1935 1919 0.0453048 0 - 0.363217 2024-04-16T08:33:02.218 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2648 16 1935 1919 0.0452876 0 - 0.363217 2024-04-16T08:33:03.218 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2649 16 1935 1919 0.0452705 0 - 0.363217 2024-04-16T08:33:04.219 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2650 16 1935 1919 0.0452535 0 - 0.363217 2024-04-16T08:33:05.219 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2651 16 1935 1919 0.0452364 0 - 0.363217 2024-04-16T08:33:06.219 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2652 16 1935 1919 0.0452193 0 - 0.363217 2024-04-16T08:33:07.219 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2653 16 1935 1919 0.0452023 0 - 0.363217 2024-04-16T08:33:08.219 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2654 16 1935 1919 0.0451853 0 - 0.363217 2024-04-16T08:33:09.219 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2655 16 1935 1919 0.0451682 0 - 0.363217 2024-04-16T08:33:10.219 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2656 16 1935 1919 0.0451512 0 - 0.363217 2024-04-16T08:33:11.219 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2657 16 1935 1919 0.0451342 0 - 0.363217 2024-04-16T08:33:12.220 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2658 16 1935 1919 0.0451173 0 - 0.363217 2024-04-16T08:33:13.222 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2659 16 1935 1919 0.0451003 0 - 0.363217 2024-04-16T08:33:14.220 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:33:14.222015+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:33:14.220 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:33:14.220 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2660 16 1935 1919 0.0450833 0 - 0.363217 2024-04-16T08:33:15.220 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2661 16 1935 1919 0.0450664 0 - 0.363217 2024-04-16T08:33:16.220 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2662 16 1935 1919 0.0450495 0 - 0.363217 2024-04-16T08:33:17.221 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2663 16 1935 1919 0.0450326 0 - 0.363217 2024-04-16T08:33:18.221 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2664 16 1935 1919 0.0450156 0 - 0.363217 2024-04-16T08:33:19.221 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2665 16 1935 1919 0.0449988 0 - 0.363217 2024-04-16T08:33:20.221 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2666 16 1935 1919 0.0449819 0 - 0.363217 2024-04-16T08:33:21.221 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2667 16 1935 1919 0.044965 0 - 0.363217 2024-04-16T08:33:22.221 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2668 16 1935 1919 0.0449482 0 - 0.363217 2024-04-16T08:33:23.221 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2669 16 1935 1919 0.0449313 0 - 0.363217 2024-04-16T08:33:24.221 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2670 16 1935 1919 0.0449145 0 - 0.363217 2024-04-16T08:33:25.222 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2671 16 1935 1919 0.0448977 0 - 0.363217 2024-04-16T08:33:26.222 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2672 16 1935 1919 0.0448809 0 - 0.363217 2024-04-16T08:33:27.222 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2673 16 1935 1919 0.0448641 0 - 0.363217 2024-04-16T08:33:28.222 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2674 16 1935 1919 0.0448473 0 - 0.363217 2024-04-16T08:33:29.222 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2675 16 1935 1919 0.0448305 0 - 0.363217 2024-04-16T08:33:30.222 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2676 16 1935 1919 0.0448138 0 - 0.363217 2024-04-16T08:33:31.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2677 16 1935 1919 0.044797 0 - 0.363217 2024-04-16T08:33:32.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2678 16 1935 1919 0.0447803 0 - 0.363217 2024-04-16T08:33:33.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2679 16 1935 1919 0.0447636 0 - 0.363217 2024-04-16T08:33:34.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:33:34.224777+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:33:34.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:33:34.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2680 16 1935 1919 0.0447469 0 - 0.363217 2024-04-16T08:33:35.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2681 16 1935 1919 0.0447302 0 - 0.363217 2024-04-16T08:33:36.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2682 16 1935 1919 0.0447135 0 - 0.363217 2024-04-16T08:33:37.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2683 16 1935 1919 0.0446969 0 - 0.363217 2024-04-16T08:33:38.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2684 16 1935 1919 0.0446802 0 - 0.363217 2024-04-16T08:33:39.223 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2685 16 1935 1919 0.0446636 0 - 0.363217 2024-04-16T08:33:40.224 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2686 16 1935 1919 0.0446469 0 - 0.363217 2024-04-16T08:33:41.224 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2687 16 1935 1919 0.0446303 0 - 0.363217 2024-04-16T08:33:42.224 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2688 16 1935 1919 0.0446137 0 - 0.363217 2024-04-16T08:33:43.224 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2689 16 1935 1919 0.0445971 0 - 0.363217 2024-04-16T08:33:44.224 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2690 16 1935 1919 0.0445805 0 - 0.363217 2024-04-16T08:33:45.224 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2691 16 1935 1919 0.044564 0 - 0.363217 2024-04-16T08:33:46.224 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2692 16 1935 1919 0.0445474 0 - 0.363217 2024-04-16T08:33:47.224 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2693 16 1935 1919 0.0445309 0 - 0.363217 2024-04-16T08:33:48.225 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2694 16 1935 1919 0.0445144 0 - 0.363217 2024-04-16T08:33:49.225 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2695 16 1935 1919 0.0444978 0 - 0.363217 2024-04-16T08:33:50.225 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2696 16 1935 1919 0.0444813 0 - 0.363217 2024-04-16T08:33:51.225 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2697 16 1935 1919 0.0444648 0 - 0.363217 2024-04-16T08:33:52.225 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2698 16 1935 1919 0.0444484 0 - 0.363217 2024-04-16T08:33:53.225 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2699 16 1935 1919 0.0444319 0 - 0.363217 2024-04-16T08:33:54.225 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:33:54.227307+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:33:54.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:33:54.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2700 16 1935 1919 0.0444154 0 - 0.363217 2024-04-16T08:33:55.225 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2701 16 1935 1919 0.044399 0 - 0.363217 2024-04-16T08:33:56.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2702 16 1935 1919 0.0443826 0 - 0.363217 2024-04-16T08:33:57.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2703 16 1935 1919 0.0443661 0 - 0.363217 2024-04-16T08:33:58.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2704 16 1935 1919 0.0443497 0 - 0.363217 2024-04-16T08:33:59.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2705 16 1935 1919 0.0443333 0 - 0.363217 2024-04-16T08:34:00.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2706 16 1935 1919 0.044317 0 - 0.363217 2024-04-16T08:34:01.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2707 16 1935 1919 0.0443006 0 - 0.363217 2024-04-16T08:34:02.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2708 16 1935 1919 0.0442842 0 - 0.363217 2024-04-16T08:34:03.226 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2709 16 1935 1919 0.0442679 0 - 0.363217 2024-04-16T08:34:04.227 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2710 16 1935 1919 0.0442515 0 - 0.363217 2024-04-16T08:34:05.227 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2711 16 1935 1919 0.0442352 0 - 0.363217 2024-04-16T08:34:06.227 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2712 16 1935 1919 0.0442189 0 - 0.363217 2024-04-16T08:34:07.227 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2713 16 1935 1919 0.0442026 0 - 0.363217 2024-04-16T08:34:08.227 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2714 16 1935 1919 0.0441863 0 - 0.363217 2024-04-16T08:34:09.227 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2715 16 1935 1919 0.04417 0 - 0.363217 2024-04-16T08:34:10.227 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2716 16 1935 1919 0.0441538 0 - 0.363217 2024-04-16T08:34:11.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2717 16 1935 1919 0.0441375 0 - 0.363217 2024-04-16T08:34:12.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2718 16 1935 1919 0.0441213 0 - 0.363217 2024-04-16T08:34:13.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2719 16 1935 1919 0.0441051 0 - 0.363217 2024-04-16T08:34:14.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:34:14.229927+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:34:14.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:34:14.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2720 16 1935 1919 0.0440889 0 - 0.363217 2024-04-16T08:34:15.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2721 16 1935 1919 0.0440727 0 - 0.363217 2024-04-16T08:34:16.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2722 16 1935 1919 0.0440565 0 - 0.363217 2024-04-16T08:34:17.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2723 16 1935 1919 0.0440403 0 - 0.363217 2024-04-16T08:34:18.228 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2724 16 1935 1919 0.0440241 0 - 0.363217 2024-04-16T08:34:19.229 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2725 16 1935 1919 0.044008 0 - 0.363217 2024-04-16T08:34:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:34:19.658 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:34:20.229 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2726 16 1935 1919 0.0439918 0 - 0.363217 2024-04-16T08:34:21.229 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2727 16 1935 1919 0.0439757 0 - 0.363217 2024-04-16T08:34:21.659 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:34:21.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:34:21.780+0000 7fa17799d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:34:22.229 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2728 16 1935 1919 0.0439596 0 - 0.363217 2024-04-16T08:34:23.229 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2729 16 1935 1919 0.0439435 0 - 0.363217 2024-04-16T08:34:24.229 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2730 16 1935 1919 0.0439274 0 - 0.363217 2024-04-16T08:34:25.230 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2731 16 1935 1919 0.0439113 0 - 0.363217 2024-04-16T08:34:26.230 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2732 16 1935 1919 0.0438952 0 - 0.363217 2024-04-16T08:34:27.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2733 16 1935 1919 0.0438791 0 - 0.363217 2024-04-16T08:34:28.230 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2734 16 1935 1919 0.0438631 0 - 0.363217 2024-04-16T08:34:29.230 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2735 16 1935 1919 0.043847 0 - 0.363217 2024-04-16T08:34:30.230 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2736 16 1935 1919 0.043831 0 - 0.363217 2024-04-16T08:34:31.230 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2737 16 1935 1919 0.043815 0 - 0.363217 2024-04-16T08:34:32.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2738 16 1935 1919 0.043799 0 - 0.363217 2024-04-16T08:34:33.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2739 16 1935 1919 0.043783 0 - 0.363217 2024-04-16T08:34:34.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:34:34.232685+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:34:34.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:34:34.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2740 16 1935 1919 0.043767 0 - 0.363217 2024-04-16T08:34:35.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2741 16 1935 1919 0.0437511 0 - 0.363217 2024-04-16T08:34:36.232 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2742 16 1935 1919 0.0437351 0 - 0.363217 2024-04-16T08:34:37.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2743 16 1935 1919 0.0437192 0 - 0.363217 2024-04-16T08:34:38.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2744 16 1935 1919 0.0437032 0 - 0.363217 2024-04-16T08:34:39.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2745 16 1935 1919 0.0436873 0 - 0.363217 2024-04-16T08:34:40.231 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2746 16 1935 1919 0.0436714 0 - 0.363217 2024-04-16T08:34:41.232 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2747 16 1935 1919 0.0436555 0 - 0.363217 2024-04-16T08:34:42.232 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2748 16 1935 1919 0.0436396 0 - 0.363217 2024-04-16T08:34:43.232 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2749 16 1935 1919 0.0436237 0 - 0.363217 2024-04-16T08:34:44.232 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2750 16 1935 1919 0.0436079 0 - 0.363217 2024-04-16T08:34:45.232 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2751 16 1935 1919 0.043592 0 - 0.363217 2024-04-16T08:34:46.232 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2752 16 1935 1919 0.0435762 0 - 0.363217 2024-04-16T08:34:47.232 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2753 16 1935 1919 0.0435604 0 - 0.363217 2024-04-16T08:34:48.233 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2754 16 1935 1919 0.0435445 0 - 0.363217 2024-04-16T08:34:49.233 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2755 16 1935 1919 0.0435287 0 - 0.363217 2024-04-16T08:34:50.233 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2756 16 1935 1919 0.0435129 0 - 0.363217 2024-04-16T08:34:51.233 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2757 16 1935 1919 0.0434972 0 - 0.363217 2024-04-16T08:34:52.233 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2758 16 1935 1919 0.0434814 0 - 0.363217 2024-04-16T08:34:53.233 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2759 16 1935 1919 0.0434656 0 - 0.363217 2024-04-16T08:34:54.233 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:34:54.235406+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:34:54.234 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:34:54.234 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2760 16 1935 1919 0.0434499 0 - 0.363217 2024-04-16T08:34:55.234 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2761 16 1935 1919 0.0434341 0 - 0.363217 2024-04-16T08:34:56.234 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2762 16 1935 1919 0.0434184 0 - 0.363217 2024-04-16T08:34:57.234 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2763 16 1935 1919 0.0434027 0 - 0.363217 2024-04-16T08:34:58.234 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2764 16 1935 1919 0.043387 0 - 0.363217 2024-04-16T08:34:59.234 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2765 16 1935 1919 0.0433713 0 - 0.363217 2024-04-16T08:35:00.234 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2766 16 1935 1919 0.0433556 0 - 0.363217 2024-04-16T08:35:01.234 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2767 16 1935 1919 0.04334 0 - 0.363217 2024-04-16T08:35:02.235 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2768 16 1935 1919 0.0433243 0 - 0.363217 2024-04-16T08:35:03.235 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2769 16 1935 1919 0.0433087 0 - 0.363217 2024-04-16T08:35:04.235 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2770 16 1935 1919 0.043293 0 - 0.363217 2024-04-16T08:35:05.235 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2771 16 1935 1919 0.0432774 0 - 0.363217 2024-04-16T08:35:06.235 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2772 16 1935 1919 0.0432618 0 - 0.363217 2024-04-16T08:35:07.235 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2773 16 1935 1919 0.0432462 0 - 0.363217 2024-04-16T08:35:08.235 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2774 16 1935 1919 0.0432306 0 - 0.363217 2024-04-16T08:35:09.236 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2775 16 1935 1919 0.043215 0 - 0.363217 2024-04-16T08:35:10.235 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2776 16 1935 1919 0.0431994 0 - 0.363217 2024-04-16T08:35:11.236 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2777 16 1935 1919 0.0431839 0 - 0.363217 2024-04-16T08:35:12.236 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2778 16 1935 1919 0.0431683 0 - 0.363217 2024-04-16T08:35:13.236 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2779 16 1935 1919 0.0431528 0 - 0.363217 2024-04-16T08:35:14.236 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:35:14.238147+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:35:14.236 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:35:14.236 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2780 16 1935 1919 0.0431373 0 - 0.363217 2024-04-16T08:35:15.236 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2781 16 1935 1919 0.0431218 0 - 0.363217 2024-04-16T08:35:16.237 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2782 16 1935 1919 0.0431063 0 - 0.363217 2024-04-16T08:35:17.236 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2783 16 1935 1919 0.0430908 0 - 0.363217 2024-04-16T08:35:18.237 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2784 16 1935 1919 0.0430753 0 - 0.363217 2024-04-16T08:35:19.237 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2785 16 1935 1919 0.0430598 0 - 0.363217 2024-04-16T08:35:20.237 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2786 16 1935 1919 0.0430444 0 - 0.363217 2024-04-16T08:35:21.237 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2787 16 1935 1919 0.0430289 0 - 0.363217 2024-04-16T08:35:22.237 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2788 16 1935 1919 0.0430135 0 - 0.363217 2024-04-16T08:35:23.237 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2789 16 1935 1919 0.0429981 0 - 0.363217 2024-04-16T08:35:24.237 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2790 16 1935 1919 0.0429827 0 - 0.363217 2024-04-16T08:35:25.238 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2791 16 1935 1919 0.0429673 0 - 0.363217 2024-04-16T08:35:26.238 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2792 16 1935 1919 0.0429519 0 - 0.363217 2024-04-16T08:35:27.238 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2793 16 1935 1919 0.0429365 0 - 0.363217 2024-04-16T08:35:28.238 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2794 16 1935 1919 0.0429211 0 - 0.363217 2024-04-16T08:35:29.238 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2795 16 1935 1919 0.0429058 0 - 0.363217 2024-04-16T08:35:30.238 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2796 16 1935 1919 0.0428904 0 - 0.363217 2024-04-16T08:35:31.238 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2797 16 1935 1919 0.0428751 0 - 0.363217 2024-04-16T08:35:32.238 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2798 16 1935 1919 0.0428598 0 - 0.363217 2024-04-16T08:35:33.239 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2799 16 1935 1919 0.0428445 0 - 0.363217 2024-04-16T08:35:34.239 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:35:34.240845+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:35:34.239 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:35:34.239 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2800 16 1935 1919 0.0428292 0 - 0.363217 2024-04-16T08:35:35.239 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2801 16 1935 1919 0.0428139 0 - 0.363217 2024-04-16T08:35:36.239 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2802 16 1935 1919 0.0427986 0 - 0.363217 2024-04-16T08:35:37.239 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2803 16 1935 1919 0.0427833 0 - 0.363217 2024-04-16T08:35:38.240 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2804 16 1935 1919 0.0427681 0 - 0.363217 2024-04-16T08:35:39.241 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2805 16 1935 1919 0.0427528 0 - 0.363217 2024-04-16T08:35:40.240 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2806 16 1935 1919 0.0427376 0 - 0.363217 2024-04-16T08:35:41.240 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2807 16 1935 1919 0.0427224 0 - 0.363217 2024-04-16T08:35:42.240 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2808 16 1935 1919 0.0427071 0 - 0.363217 2024-04-16T08:35:43.240 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2809 16 1935 1919 0.0426919 0 - 0.363217 2024-04-16T08:35:44.240 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2810 16 1935 1919 0.0426767 0 - 0.363217 2024-04-16T08:35:45.240 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2811 16 1935 1919 0.0426616 0 - 0.363217 2024-04-16T08:35:46.241 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2812 16 1935 1919 0.0426464 0 - 0.363217 2024-04-16T08:35:47.241 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2813 16 1935 1919 0.0426312 0 - 0.363217 2024-04-16T08:35:48.241 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2814 16 1935 1919 0.0426161 0 - 0.363217 2024-04-16T08:35:49.241 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2815 16 1935 1919 0.0426009 0 - 0.363217 2024-04-16T08:35:50.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2816 16 1935 1919 0.0425858 0 - 0.363217 2024-04-16T08:35:51.241 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2817 16 1935 1919 0.0425707 0 - 0.363217 2024-04-16T08:35:52.241 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2818 16 1935 1919 0.0425556 0 - 0.363217 2024-04-16T08:35:53.242 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2819 16 1935 1919 0.0425405 0 - 0.363217 2024-04-16T08:35:54.242 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:35:54.243791+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:35:54.242 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:35:54.242 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2820 16 1935 1919 0.0425254 0 - 0.363217 2024-04-16T08:35:55.242 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2821 16 1935 1919 0.0425103 0 - 0.363217 2024-04-16T08:35:56.242 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2822 16 1935 1919 0.0424953 0 - 0.363217 2024-04-16T08:35:57.242 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2823 16 1935 1919 0.0424802 0 - 0.363217 2024-04-16T08:35:58.242 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2824 16 1935 1919 0.0424652 0 - 0.363217 2024-04-16T08:35:59.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2825 16 1935 1919 0.0424501 0 - 0.363217 2024-04-16T08:36:00.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2826 16 1935 1919 0.0424351 0 - 0.363217 2024-04-16T08:36:01.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2827 16 1935 1919 0.0424201 0 - 0.363217 2024-04-16T08:36:02.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2828 16 1935 1919 0.0424051 0 - 0.363217 2024-04-16T08:36:03.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2829 16 1935 1919 0.0423901 0 - 0.363217 2024-04-16T08:36:04.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2830 16 1935 1919 0.0423751 0 - 0.363217 2024-04-16T08:36:05.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2831 16 1935 1919 0.0423602 0 - 0.363217 2024-04-16T08:36:06.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2832 16 1935 1919 0.0423452 0 - 0.363217 2024-04-16T08:36:07.243 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2833 16 1935 1919 0.0423303 0 - 0.363217 2024-04-16T08:36:08.244 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2834 16 1935 1919 0.0423153 0 - 0.363217 2024-04-16T08:36:09.244 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2835 16 1935 1919 0.0423004 0 - 0.363217 2024-04-16T08:36:10.244 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2836 16 1935 1919 0.0422855 0 - 0.363217 2024-04-16T08:36:11.244 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2837 16 1935 1919 0.0422706 0 - 0.363217 2024-04-16T08:36:12.244 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2838 16 1935 1919 0.0422557 0 - 0.363217 2024-04-16T08:36:13.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2839 16 1935 1919 0.0422408 0 - 0.363217 2024-04-16T08:36:14.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:36:14.246595+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:36:14.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:36:14.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2840 16 1935 1919 0.0422259 0 - 0.363217 2024-04-16T08:36:15.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2841 16 1935 1919 0.0422111 0 - 0.363217 2024-04-16T08:36:16.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2842 16 1935 1919 0.0421962 0 - 0.363217 2024-04-16T08:36:17.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2843 16 1935 1919 0.0421814 0 - 0.363217 2024-04-16T08:36:18.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2844 16 1935 1919 0.0421665 0 - 0.363217 2024-04-16T08:36:19.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2845 16 1935 1919 0.0421517 0 - 0.363217 2024-04-16T08:36:20.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2846 16 1935 1919 0.0421369 0 - 0.363217 2024-04-16T08:36:21.245 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2847 16 1935 1919 0.0421221 0 - 0.363217 2024-04-16T08:36:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:36:21.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:36:22.246 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2848 16 1935 1919 0.0421073 0 - 0.363217 2024-04-16T08:36:23.246 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2849 16 1935 1919 0.0420925 0 - 0.363217 2024-04-16T08:36:23.700 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-16T08:36:23.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:36:23.823+0000 7f8657f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:36:24.246 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2850 16 1935 1919 0.0420778 0 - 0.363217 2024-04-16T08:36:25.246 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2851 16 1935 1919 0.042063 0 - 0.363217 2024-04-16T08:36:26.246 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2852 16 1935 1919 0.0420483 0 - 0.363217 2024-04-16T08:36:27.246 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2853 16 1935 1919 0.0420335 0 - 0.363217 2024-04-16T08:36:28.246 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2854 16 1935 1919 0.0420188 0 - 0.363217 2024-04-16T08:36:29.247 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2855 16 1935 1919 0.0420041 0 - 0.363217 2024-04-16T08:36:30.247 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2856 16 1935 1919 0.0419894 0 - 0.363217 2024-04-16T08:36:31.247 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2857 16 1935 1919 0.0419747 0 - 0.363217 2024-04-16T08:36:32.247 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2858 16 1935 1919 0.04196 0 - 0.363217 2024-04-16T08:36:33.247 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2859 16 1935 1919 0.0419453 0 - 0.363217 2024-04-16T08:36:34.247 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:36:34.249298+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:36:34.248 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:36:34.248 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2860 16 1935 1919 0.0419306 0 - 0.363217 2024-04-16T08:36:35.247 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2861 16 1935 1919 0.041916 0 - 0.363217 2024-04-16T08:36:36.247 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2862 16 1935 1919 0.0419013 0 - 0.363217 2024-04-16T08:36:37.248 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2863 16 1935 1919 0.0418867 0 - 0.363217 2024-04-16T08:36:38.248 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2864 16 1935 1919 0.0418721 0 - 0.363217 2024-04-16T08:36:39.248 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2865 16 1935 1919 0.0418575 0 - 0.363217 2024-04-16T08:36:40.248 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2866 16 1935 1919 0.0418429 0 - 0.363217 2024-04-16T08:36:41.248 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2867 16 1935 1919 0.0418283 0 - 0.363217 2024-04-16T08:36:42.248 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2868 16 1935 1919 0.0418137 0 - 0.363217 2024-04-16T08:36:43.249 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2869 16 1935 1919 0.0417991 0 - 0.363217 2024-04-16T08:36:44.249 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2870 16 1935 1919 0.0417845 0 - 0.363217 2024-04-16T08:36:45.249 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2871 16 1935 1919 0.04177 0 - 0.363217 2024-04-16T08:36:46.249 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2872 16 1935 1919 0.0417554 0 - 0.363217 2024-04-16T08:36:47.249 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2873 16 1935 1919 0.0417409 0 - 0.363217 2024-04-16T08:36:48.249 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2874 16 1935 1919 0.0417264 0 - 0.363217 2024-04-16T08:36:49.249 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2875 16 1935 1919 0.0417119 0 - 0.363217 2024-04-16T08:36:50.249 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2876 16 1935 1919 0.0416974 0 - 0.363217 2024-04-16T08:36:51.250 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2877 16 1935 1919 0.0416829 0 - 0.363217 2024-04-16T08:36:52.250 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2878 16 1935 1919 0.0416684 0 - 0.363217 2024-04-16T08:36:53.250 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2879 16 1935 1919 0.0416539 0 - 0.363217 2024-04-16T08:36:54.250 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:36:54.251961+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:36:54.250 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:36:54.250 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2880 16 1935 1919 0.0416395 0 - 0.363217 2024-04-16T08:36:55.250 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2881 16 1935 1919 0.041625 0 - 0.363217 2024-04-16T08:36:56.250 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2882 16 1935 1919 0.0416106 0 - 0.363217 2024-04-16T08:36:57.250 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2883 16 1935 1919 0.0415961 0 - 0.363217 2024-04-16T08:36:58.251 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2884 16 1935 1919 0.0415817 0 - 0.363217 2024-04-16T08:36:59.251 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2885 16 1935 1919 0.0415673 0 - 0.363217 2024-04-16T08:37:00.251 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2886 16 1935 1919 0.0415529 0 - 0.363217 2024-04-16T08:37:01.251 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2887 16 1935 1919 0.0415385 0 - 0.363217 2024-04-16T08:37:02.251 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2888 16 1935 1919 0.0415241 0 - 0.363217 2024-04-16T08:37:03.251 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2889 16 1935 1919 0.0415097 0 - 0.363217 2024-04-16T08:37:04.251 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2890 16 1935 1919 0.0414954 0 - 0.363217 2024-04-16T08:37:05.251 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2891 16 1935 1919 0.041481 0 - 0.363217 2024-04-16T08:37:06.252 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2892 16 1935 1919 0.0414667 0 - 0.363217 2024-04-16T08:37:07.252 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2893 16 1935 1919 0.0414523 0 - 0.363217 2024-04-16T08:37:08.252 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2894 16 1935 1919 0.041438 0 - 0.363217 2024-04-16T08:37:09.252 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2895 16 1935 1919 0.0414237 0 - 0.363217 2024-04-16T08:37:10.252 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2896 16 1935 1919 0.0414094 0 - 0.363217 2024-04-16T08:37:11.253 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2897 16 1935 1919 0.0413951 0 - 0.363217 2024-04-16T08:37:12.252 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2898 16 1935 1919 0.0413808 0 - 0.363217 2024-04-16T08:37:13.253 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2899 16 1935 1919 0.0413666 0 - 0.363217 2024-04-16T08:37:14.253 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:37:14.254714+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:37:14.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:37:14.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2900 16 1935 1919 0.0413523 0 - 0.363217 2024-04-16T08:37:15.253 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2901 16 1935 1919 0.041338 0 - 0.363217 2024-04-16T08:37:16.253 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2902 16 1935 1919 0.0413238 0 - 0.363217 2024-04-16T08:37:17.253 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2903 16 1935 1919 0.0413096 0 - 0.363217 2024-04-16T08:37:18.253 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2904 16 1935 1919 0.0412953 0 - 0.363217 2024-04-16T08:37:19.253 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2905 16 1935 1919 0.0412811 0 - 0.363217 2024-04-16T08:37:20.253 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2906 16 1935 1919 0.0412669 0 - 0.363217 2024-04-16T08:37:21.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2907 16 1935 1919 0.0412527 0 - 0.363217 2024-04-16T08:37:22.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2908 16 1935 1919 0.0412385 0 - 0.363217 2024-04-16T08:37:23.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2909 16 1935 1919 0.0412244 0 - 0.363217 2024-04-16T08:37:24.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2910 16 1935 1919 0.0412102 0 - 0.363217 2024-04-16T08:37:25.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2911 16 1935 1919 0.041196 0 - 0.363217 2024-04-16T08:37:26.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2912 16 1935 1919 0.0411819 0 - 0.363217 2024-04-16T08:37:27.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2913 16 1935 1919 0.0411677 0 - 0.363217 2024-04-16T08:37:28.254 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2914 16 1935 1919 0.0411536 0 - 0.363217 2024-04-16T08:37:29.255 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2915 16 1935 1919 0.0411395 0 - 0.363217 2024-04-16T08:37:30.255 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2916 16 1935 1919 0.0411254 0 - 0.363217 2024-04-16T08:37:31.255 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2917 16 1935 1919 0.0411113 0 - 0.363217 2024-04-16T08:37:32.255 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2918 16 1935 1919 0.0410972 0 - 0.363217 2024-04-16T08:37:33.255 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2919 16 1935 1919 0.0410831 0 - 0.363217 2024-04-16T08:37:34.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:37:34.257457+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:37:34.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:37:34.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2920 16 1935 1919 0.0410691 0 - 0.363217 2024-04-16T08:37:35.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2921 16 1935 1919 0.041055 0 - 0.363217 2024-04-16T08:37:36.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2922 16 1935 1919 0.0410409 0 - 0.363217 2024-04-16T08:37:37.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2923 16 1935 1919 0.0410269 0 - 0.363217 2024-04-16T08:37:38.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2924 16 1935 1919 0.0410129 0 - 0.363217 2024-04-16T08:37:39.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2925 16 1935 1919 0.0409988 0 - 0.363217 2024-04-16T08:37:40.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2926 16 1935 1919 0.0409848 0 - 0.363217 2024-04-16T08:37:41.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2927 16 1935 1919 0.0409708 0 - 0.363217 2024-04-16T08:37:42.256 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2928 16 1935 1919 0.0409568 0 - 0.363217 2024-04-16T08:37:43.257 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2929 16 1935 1919 0.0409429 0 - 0.363217 2024-04-16T08:37:44.257 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2930 16 1935 1919 0.0409289 0 - 0.363217 2024-04-16T08:37:45.257 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2931 16 1935 1919 0.0409149 0 - 0.363217 2024-04-16T08:37:46.257 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2932 16 1935 1919 0.040901 0 - 0.363217 2024-04-16T08:37:47.257 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2933 16 1935 1919 0.040887 0 - 0.363217 2024-04-16T08:37:48.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2934 16 1935 1919 0.0408731 0 - 0.363217 2024-04-16T08:37:49.258 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2935 16 1935 1919 0.0408592 0 - 0.363217 2024-04-16T08:37:50.258 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2936 16 1935 1919 0.0408452 0 - 0.363217 2024-04-16T08:37:51.258 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2937 16 1935 1919 0.0408313 0 - 0.363217 2024-04-16T08:37:52.258 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2938 16 1935 1919 0.0408174 0 - 0.363217 2024-04-16T08:37:53.258 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2939 16 1935 1919 0.0408035 0 - 0.363217 2024-04-16T08:37:54.258 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:37:54.260200+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:37:54.258 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:37:54.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2940 16 1935 1919 0.0407897 0 - 0.363217 2024-04-16T08:37:55.258 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2941 16 1935 1919 0.0407758 0 - 0.363217 2024-04-16T08:37:56.262 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2942 16 1935 1919 0.0407619 0 - 0.363217 2024-04-16T08:37:57.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2943 16 1935 1919 0.0407481 0 - 0.363217 2024-04-16T08:37:58.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2944 16 1935 1919 0.0407342 0 - 0.363217 2024-04-16T08:37:59.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2945 16 1935 1919 0.0407204 0 - 0.363217 2024-04-16T08:38:00.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2946 16 1935 1919 0.0407066 0 - 0.363217 2024-04-16T08:38:01.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2947 16 1935 1919 0.0406928 0 - 0.363217 2024-04-16T08:38:02.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2948 16 1935 1919 0.040679 0 - 0.363217 2024-04-16T08:38:03.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2949 16 1935 1919 0.0406652 0 - 0.363217 2024-04-16T08:38:04.259 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2950 16 1935 1919 0.0406514 0 - 0.363217 2024-04-16T08:38:05.260 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2951 16 1935 1919 0.0406376 0 - 0.363217 2024-04-16T08:38:06.260 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2952 16 1935 1919 0.0406239 0 - 0.363217 2024-04-16T08:38:07.260 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2953 16 1935 1919 0.0406101 0 - 0.363217 2024-04-16T08:38:08.260 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2954 16 1935 1919 0.0405964 0 - 0.363217 2024-04-16T08:38:09.260 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2955 16 1935 1919 0.0405826 0 - 0.363217 2024-04-16T08:38:10.260 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2956 16 1935 1919 0.0405689 0 - 0.363217 2024-04-16T08:38:11.260 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2957 16 1935 1919 0.0405552 0 - 0.363217 2024-04-16T08:38:12.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2958 16 1935 1919 0.0405415 0 - 0.363217 2024-04-16T08:38:13.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2959 16 1935 1919 0.0405278 0 - 0.363217 2024-04-16T08:38:14.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:38:14.262845+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:38:14.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:38:14.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2960 16 1935 1919 0.0405141 0 - 0.363217 2024-04-16T08:38:15.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2961 16 1935 1919 0.0405004 0 - 0.363217 2024-04-16T08:38:16.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2962 16 1935 1919 0.0404867 0 - 0.363217 2024-04-16T08:38:17.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2963 16 1935 1919 0.040473 0 - 0.363217 2024-04-16T08:38:18.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2964 16 1935 1919 0.0404594 0 - 0.363217 2024-04-16T08:38:19.261 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2965 16 1935 1919 0.0404457 0 - 0.363217 2024-04-16T08:38:20.262 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2966 16 1935 1919 0.0404321 0 - 0.363217 2024-04-16T08:38:21.262 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2967 16 1935 1919 0.0404185 0 - 0.363217 2024-04-16T08:38:22.262 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2968 16 1935 1919 0.0404049 0 - 0.363217 2024-04-16T08:38:23.262 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2969 16 1935 1919 0.0403913 0 - 0.363217 2024-04-16T08:38:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:38:23.741 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:38:24.262 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2970 16 1935 1919 0.0403777 0 - 0.363217 2024-04-16T08:38:25.262 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2971 16 1935 1919 0.0403641 0 - 0.363217 2024-04-16T08:38:25.743 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-16T08:38:25.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:38:25.864+0000 7f2a6acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:38:26.262 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2972 16 1935 1919 0.0403505 0 - 0.363217 2024-04-16T08:38:27.263 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2973 16 1935 1919 0.0403369 0 - 0.363217 2024-04-16T08:38:28.263 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2974 16 1935 1919 0.0403233 0 - 0.363217 2024-04-16T08:38:29.263 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2975 16 1935 1919 0.0403098 0 - 0.363217 2024-04-16T08:38:30.263 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2976 16 1935 1919 0.0402962 0 - 0.363217 2024-04-16T08:38:31.263 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2977 16 1935 1919 0.0402827 0 - 0.363217 2024-04-16T08:38:32.263 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2978 16 1935 1919 0.0402692 0 - 0.363217 2024-04-16T08:38:33.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2979 16 1935 1919 0.0402557 0 - 0.363217 2024-04-16T08:38:34.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:38:34.265612+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:38:34.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:38:34.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2980 16 1935 1919 0.0402422 0 - 0.363217 2024-04-16T08:38:35.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2981 16 1935 1919 0.0402287 0 - 0.363217 2024-04-16T08:38:36.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2982 16 1935 1919 0.0402152 0 - 0.363217 2024-04-16T08:38:37.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2983 16 1935 1919 0.0402017 0 - 0.363217 2024-04-16T08:38:38.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2984 16 1935 1919 0.0401882 0 - 0.363217 2024-04-16T08:38:39.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2985 16 1935 1919 0.0401747 0 - 0.363217 2024-04-16T08:38:40.264 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2986 16 1935 1919 0.0401613 0 - 0.363217 2024-04-16T08:38:41.265 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2987 16 1935 1919 0.0401478 0 - 0.363217 2024-04-16T08:38:42.265 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2988 16 1935 1919 0.0401344 0 - 0.363217 2024-04-16T08:38:43.265 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2989 16 1935 1919 0.040121 0 - 0.363217 2024-04-16T08:38:44.265 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2990 16 1935 1919 0.0401076 0 - 0.363217 2024-04-16T08:38:45.265 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2991 16 1935 1919 0.0400942 0 - 0.363217 2024-04-16T08:38:46.265 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2992 16 1935 1919 0.0400808 0 - 0.363217 2024-04-16T08:38:47.265 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2993 16 1935 1919 0.0400674 0 - 0.363217 2024-04-16T08:38:48.266 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2994 16 1935 1919 0.040054 0 - 0.363217 2024-04-16T08:38:49.266 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2995 16 1935 1919 0.0400406 0 - 0.363217 2024-04-16T08:38:50.266 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2996 16 1935 1919 0.0400272 0 - 0.363217 2024-04-16T08:38:51.266 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2997 16 1935 1919 0.0400139 0 - 0.363217 2024-04-16T08:38:52.266 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2998 16 1935 1919 0.0400005 0 - 0.363217 2024-04-16T08:38:53.266 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 2999 16 1935 1919 0.0399872 0 - 0.363217 2024-04-16T08:38:54.266 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:38:54.268292+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:38:54.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:38:54.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3000 16 1935 1919 0.0399739 0 - 0.363217 2024-04-16T08:38:55.266 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3001 16 1935 1919 0.0399606 0 - 0.363217 2024-04-16T08:38:56.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3002 16 1935 1919 0.0399472 0 - 0.363217 2024-04-16T08:38:57.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3003 16 1935 1919 0.0399339 0 - 0.363217 2024-04-16T08:38:58.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3004 16 1935 1919 0.0399206 0 - 0.363217 2024-04-16T08:38:59.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3005 16 1935 1919 0.0399074 0 - 0.363217 2024-04-16T08:39:00.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3006 16 1935 1919 0.0398941 0 - 0.363217 2024-04-16T08:39:01.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3007 16 1935 1919 0.0398808 0 - 0.363217 2024-04-16T08:39:02.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3008 16 1935 1919 0.0398676 0 - 0.363217 2024-04-16T08:39:03.267 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3009 16 1935 1919 0.0398543 0 - 0.363217 2024-04-16T08:39:04.268 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3010 16 1935 1919 0.0398411 0 - 0.363217 2024-04-16T08:39:05.268 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3011 16 1935 1919 0.0398278 0 - 0.363217 2024-04-16T08:39:06.268 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3012 16 1935 1919 0.0398146 0 - 0.363217 2024-04-16T08:39:07.268 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3013 16 1935 1919 0.0398014 0 - 0.363217 2024-04-16T08:39:08.268 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3014 16 1935 1919 0.0397882 0 - 0.363217 2024-04-16T08:39:09.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3015 16 1935 1919 0.039775 0 - 0.363217 2024-04-16T08:39:10.268 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3016 16 1935 1919 0.0397618 0 - 0.363217 2024-04-16T08:39:11.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3017 16 1935 1919 0.0397486 0 - 0.363217 2024-04-16T08:39:12.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3018 16 1935 1919 0.0397355 0 - 0.363217 2024-04-16T08:39:13.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3019 16 1935 1919 0.0397223 0 - 0.363217 2024-04-16T08:39:14.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:39:14.270959+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:39:14.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:39:14.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3020 16 1935 1919 0.0397091 0 - 0.363217 2024-04-16T08:39:15.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3021 16 1935 1919 0.039696 0 - 0.363217 2024-04-16T08:39:16.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3022 16 1935 1919 0.0396829 0 - 0.363217 2024-04-16T08:39:17.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3023 16 1935 1919 0.0396697 0 - 0.363217 2024-04-16T08:39:18.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3024 16 1935 1919 0.0396566 0 - 0.363217 2024-04-16T08:39:19.269 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3025 16 1935 1919 0.0396435 0 - 0.363217 2024-04-16T08:39:20.270 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3026 16 1935 1919 0.0396304 0 - 0.363217 2024-04-16T08:39:21.270 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3027 16 1935 1919 0.0396173 0 - 0.363217 2024-04-16T08:39:22.270 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3028 16 1935 1919 0.0396042 0 - 0.363217 2024-04-16T08:39:23.270 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3029 16 1935 1919 0.0395912 0 - 0.363217 2024-04-16T08:39:24.270 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3030 16 1935 1919 0.0395781 0 - 0.363217 2024-04-16T08:39:25.271 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3031 16 1935 1919 0.039565 0 - 0.363217 2024-04-16T08:39:26.271 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3032 16 1935 1919 0.039552 0 - 0.363217 2024-04-16T08:39:27.271 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3033 16 1935 1919 0.0395389 0 - 0.363217 2024-04-16T08:39:28.271 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3034 16 1935 1919 0.0395259 0 - 0.363217 2024-04-16T08:39:29.271 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3035 16 1935 1919 0.0395129 0 - 0.363217 2024-04-16T08:39:30.271 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3036 16 1935 1919 0.0394999 0 - 0.363217 2024-04-16T08:39:31.271 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3037 16 1935 1919 0.0394869 0 - 0.363217 2024-04-16T08:39:32.271 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3038 16 1935 1919 0.0394739 0 - 0.363217 2024-04-16T08:39:33.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3039 16 1935 1919 0.0394609 0 - 0.363217 2024-04-16T08:39:34.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:39:34.273680+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:39:34.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:39:34.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3040 16 1935 1919 0.0394479 0 - 0.363217 2024-04-16T08:39:35.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3041 16 1935 1919 0.0394349 0 - 0.363217 2024-04-16T08:39:36.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3042 16 1935 1919 0.039422 0 - 0.363217 2024-04-16T08:39:37.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3043 16 1935 1919 0.039409 0 - 0.363217 2024-04-16T08:39:38.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3044 16 1935 1919 0.0393961 0 - 0.363217 2024-04-16T08:39:38.498 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (501) after waiting for 3000 seconds 2024-04-16T08:39:38.713 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=084a54f686b64bfe9bc6f7f885aa0dc2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (501) after waiting for 3000 seconds 2024-04-16T08:39:38.715 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-16T08:39:38.726 INFO:tasks.thrashosds:joining thrashosds 2024-04-16T08:39:38.726 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-16T08:39:39.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3045 16 1935 1919 0.0393831 0 - 0.363217 2024-04-16T08:39:40.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3046 16 1935 1919 0.0393702 0 - 0.363217 2024-04-16T08:39:41.272 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3047 16 1935 1919 0.0393573 0 - 0.363217 2024-04-16T08:39:42.273 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3048 16 1935 1919 0.0393444 0 - 0.363217 2024-04-16T08:39:43.273 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3049 16 1935 1919 0.0393315 0 - 0.363217 2024-04-16T08:39:44.273 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3050 16 1935 1919 0.0393186 0 - 0.363217 2024-04-16T08:39:45.273 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3051 16 1935 1919 0.0393057 0 - 0.363217 2024-04-16T08:39:46.273 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3052 16 1935 1919 0.0392928 0 - 0.363217 2024-04-16T08:39:47.273 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3053 16 1935 1919 0.0392799 0 - 0.363217 2024-04-16T08:39:48.273 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3054 16 1935 1919 0.0392671 0 - 0.363217 2024-04-16T08:39:49.274 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3055 16 1935 1919 0.0392542 0 - 0.363217 2024-04-16T08:39:50.274 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3056 16 1935 1919 0.0392414 0 - 0.363217 2024-04-16T08:39:51.274 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3057 16 1935 1919 0.0392285 0 - 0.363217 2024-04-16T08:39:52.274 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3058 16 1935 1919 0.0392157 0 - 0.363217 2024-04-16T08:39:53.274 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3059 16 1935 1919 0.0392029 0 - 0.363217 2024-04-16T08:39:54.274 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:39:54.276307+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:39:54.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:39:54.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3060 16 1935 1919 0.0391901 0 - 0.363217 2024-04-16T08:39:55.274 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3061 16 1935 1919 0.0391773 0 - 0.363217 2024-04-16T08:39:56.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3062 16 1935 1919 0.0391645 0 - 0.363217 2024-04-16T08:39:57.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3063 16 1935 1919 0.0391517 0 - 0.363217 2024-04-16T08:39:58.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3064 16 1935 1919 0.0391389 0 - 0.363217 2024-04-16T08:39:59.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3065 16 1935 1919 0.0391261 0 - 0.363217 2024-04-16T08:40:00.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3066 16 1935 1919 0.0391134 0 - 0.363217 2024-04-16T08:40:01.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3067 16 1935 1919 0.0391006 0 - 0.363217 2024-04-16T08:40:02.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3068 16 1935 1919 0.0390879 0 - 0.363217 2024-04-16T08:40:03.275 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3069 16 1935 1919 0.0390751 0 - 0.363217 2024-04-16T08:40:04.276 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3070 16 1935 1919 0.0390624 0 - 0.363217 2024-04-16T08:40:05.276 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3071 16 1935 1919 0.0390497 0 - 0.363217 2024-04-16T08:40:06.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3072 16 1935 1919 0.039037 0 - 0.363217 2024-04-16T08:40:07.276 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3073 16 1935 1919 0.0390243 0 - 0.363217 2024-04-16T08:40:08.276 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3074 16 1935 1919 0.0390116 0 - 0.363217 2024-04-16T08:40:09.276 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3075 16 1935 1919 0.0389989 0 - 0.363217 2024-04-16T08:40:10.276 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3076 16 1935 1919 0.0389862 0 - 0.363217 2024-04-16T08:40:11.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3077 16 1935 1919 0.0389736 0 - 0.363217 2024-04-16T08:40:12.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3078 16 1935 1919 0.0389609 0 - 0.363217 2024-04-16T08:40:13.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3079 16 1935 1919 0.0389482 0 - 0.363217 2024-04-16T08:40:14.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:40:14.278993+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:40:14.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:40:14.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3080 16 1935 1919 0.0389356 0 - 0.363217 2024-04-16T08:40:15.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3081 16 1935 1919 0.038923 0 - 0.363217 2024-04-16T08:40:16.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3082 16 1935 1919 0.0389103 0 - 0.363217 2024-04-16T08:40:17.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3083 16 1935 1919 0.0388977 0 - 0.363217 2024-04-16T08:40:18.277 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3084 16 1935 1919 0.0388851 0 - 0.363217 2024-04-16T08:40:19.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3085 16 1935 1919 0.0388725 0 - 0.363217 2024-04-16T08:40:20.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3086 16 1935 1919 0.0388599 0 - 0.363217 2024-04-16T08:40:21.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3087 16 1935 1919 0.0388473 0 - 0.363217 2024-04-16T08:40:22.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3088 16 1935 1919 0.0388347 0 - 0.363217 2024-04-16T08:40:23.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3089 16 1935 1919 0.0388221 0 - 0.363217 2024-04-16T08:40:24.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3090 16 1935 1919 0.0388096 0 - 0.363217 2024-04-16T08:40:25.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3091 16 1935 1919 0.038797 0 - 0.363217 2024-04-16T08:40:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:40:25.783 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-16T08:40:26.278 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3092 16 1935 1919 0.0387845 0 - 0.363217 2024-04-16T08:40:27.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3093 16 1935 1919 0.0387719 0 - 0.363217 2024-04-16T08:40:27.785 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T08:40:27.785 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T08:40:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:40:27.906+0000 7fd54f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:40:28.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3094 16 1935 1919 0.0387594 0 - 0.363217 2024-04-16T08:40:29.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3095 16 1935 1919 0.0387469 0 - 0.363217 2024-04-16T08:40:30.279 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3096 16 1935 1919 0.0387344 0 - 0.363217 2024-04-16T08:40:31.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3097 16 1935 1919 0.0387219 0 - 0.363217 2024-04-16T08:40:32.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3098 16 1935 1919 0.0387094 0 - 0.363217 2024-04-16T08:40:33.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3099 16 1935 1919 0.0386969 0 - 0.363217 2024-04-16T08:40:34.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:40:34.281738+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:40:34.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:40:34.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3100 16 1935 1919 0.0386844 0 - 0.363217 2024-04-16T08:40:35.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3101 16 1935 1919 0.0386719 0 - 0.363217 2024-04-16T08:40:36.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3102 16 1935 1919 0.0386595 0 - 0.363217 2024-04-16T08:40:37.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3103 16 1935 1919 0.038647 0 - 0.363217 2024-04-16T08:40:38.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3104 16 1935 1919 0.0386345 0 - 0.363217 2024-04-16T08:40:39.280 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3105 16 1935 1919 0.0386221 0 - 0.363217 2024-04-16T08:40:40.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3106 16 1935 1919 0.0386097 0 - 0.363217 2024-04-16T08:40:41.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3107 16 1935 1919 0.0385972 0 - 0.363217 2024-04-16T08:40:42.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3108 16 1935 1919 0.0385848 0 - 0.363217 2024-04-16T08:40:43.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3109 16 1935 1919 0.0385724 0 - 0.363217 2024-04-16T08:40:44.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3110 16 1935 1919 0.03856 0 - 0.363217 2024-04-16T08:40:45.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3111 16 1935 1919 0.0385476 0 - 0.363217 2024-04-16T08:40:46.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3112 16 1935 1919 0.0385352 0 - 0.363217 2024-04-16T08:40:47.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3113 16 1935 1919 0.0385228 0 - 0.363217 2024-04-16T08:40:48.281 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3114 16 1935 1919 0.0385105 0 - 0.363217 2024-04-16T08:40:49.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3115 16 1935 1919 0.0384981 0 - 0.363217 2024-04-16T08:40:50.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3116 16 1935 1919 0.0384858 0 - 0.363217 2024-04-16T08:40:51.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3117 16 1935 1919 0.0384734 0 - 0.363217 2024-04-16T08:40:52.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3118 16 1935 1919 0.0384611 0 - 0.363217 2024-04-16T08:40:53.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3119 16 1935 1919 0.0384487 0 - 0.363217 2024-04-16T08:40:54.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:40:54.284322+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:40:54.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:40:54.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3120 16 1935 1919 0.0384364 0 - 0.363217 2024-04-16T08:40:55.282 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3121 16 1935 1919 0.0384241 0 - 0.363217 2024-04-16T08:40:56.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3122 16 1935 1919 0.0384118 0 - 0.363217 2024-04-16T08:40:57.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3123 16 1935 1919 0.0383995 0 - 0.363217 2024-04-16T08:40:58.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3124 16 1935 1919 0.0383872 0 - 0.363217 2024-04-16T08:40:59.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3125 16 1935 1919 0.0383749 0 - 0.363217 2024-04-16T08:41:00.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3126 16 1935 1919 0.0383626 0 - 0.363217 2024-04-16T08:41:01.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3127 16 1935 1919 0.0383504 0 - 0.363217 2024-04-16T08:41:02.283 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3128 16 1935 1919 0.0383381 0 - 0.363217 2024-04-16T08:41:03.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3129 16 1935 1919 0.0383259 0 - 0.363217 2024-04-16T08:41:04.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3130 16 1935 1919 0.0383136 0 - 0.363217 2024-04-16T08:41:05.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3131 16 1935 1919 0.0383014 0 - 0.363217 2024-04-16T08:41:06.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3132 16 1935 1919 0.0382892 0 - 0.363217 2024-04-16T08:41:07.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3133 16 1935 1919 0.0382769 0 - 0.363217 2024-04-16T08:41:08.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3134 16 1935 1919 0.0382647 0 - 0.363217 2024-04-16T08:41:09.284 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3135 16 1935 1919 0.0382525 0 - 0.363217 2024-04-16T08:41:10.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3136 16 1935 1919 0.0382403 0 - 0.363217 2024-04-16T08:41:11.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3137 16 1935 1919 0.0382281 0 - 0.363217 2024-04-16T08:41:12.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3138 16 1935 1919 0.0382159 0 - 0.363217 2024-04-16T08:41:13.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3139 16 1935 1919 0.0382038 0 - 0.363217 2024-04-16T08:41:14.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:41:14.287057+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:41:14.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:41:14.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3140 16 1935 1919 0.0381916 0 - 0.363217 2024-04-16T08:41:15.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3141 16 1935 1919 0.0381794 0 - 0.363217 2024-04-16T08:41:16.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3142 16 1935 1919 0.0381673 0 - 0.363217 2024-04-16T08:41:17.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3143 16 1935 1919 0.0381551 0 - 0.363217 2024-04-16T08:41:18.285 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3144 16 1935 1919 0.038143 0 - 0.363217 2024-04-16T08:41:19.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3145 16 1935 1919 0.0381309 0 - 0.363217 2024-04-16T08:41:20.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3146 16 1935 1919 0.0381188 0 - 0.363217 2024-04-16T08:41:21.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3147 16 1935 1919 0.0381066 0 - 0.363217 2024-04-16T08:41:22.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3148 16 1935 1919 0.0380945 0 - 0.363217 2024-04-16T08:41:23.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3149 16 1935 1919 0.0380824 0 - 0.363217 2024-04-16T08:41:24.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3150 16 1935 1919 0.0380704 0 - 0.363217 2024-04-16T08:41:25.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3151 16 1935 1919 0.0380583 0 - 0.363217 2024-04-16T08:41:26.286 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3152 16 1935 1919 0.0380462 0 - 0.363217 2024-04-16T08:41:27.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3153 16 1935 1919 0.0380341 0 - 0.363217 2024-04-16T08:41:28.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3154 16 1935 1919 0.0380221 0 - 0.363217 2024-04-16T08:41:29.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3155 16 1935 1919 0.03801 0 - 0.363217 2024-04-16T08:41:30.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3156 16 1935 1919 0.037998 0 - 0.363217 2024-04-16T08:41:31.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3157 16 1935 1919 0.0379859 0 - 0.363217 2024-04-16T08:41:32.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3158 16 1935 1919 0.0379739 0 - 0.363217 2024-04-16T08:41:33.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3159 16 1935 1919 0.0379619 0 - 0.363217 2024-04-16T08:41:34.287 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:41:34.289571+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:41:34.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:41:34.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3160 16 1935 1919 0.0379499 0 - 0.363217 2024-04-16T08:41:35.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3161 16 1935 1919 0.0379379 0 - 0.363217 2024-04-16T08:41:36.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3162 16 1935 1919 0.0379259 0 - 0.363217 2024-04-16T08:41:37.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3163 16 1935 1919 0.0379139 0 - 0.363217 2024-04-16T08:41:38.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3164 16 1935 1919 0.0379019 0 - 0.363217 2024-04-16T08:41:39.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3165 16 1935 1919 0.0378899 0 - 0.363217 2024-04-16T08:41:40.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3166 16 1935 1919 0.037878 0 - 0.363217 2024-04-16T08:41:41.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3167 16 1935 1919 0.037866 0 - 0.363217 2024-04-16T08:41:42.288 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3168 16 1935 1919 0.037854 0 - 0.363217 2024-04-16T08:41:43.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3169 16 1935 1919 0.0378421 0 - 0.363217 2024-04-16T08:41:44.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3170 16 1935 1919 0.0378302 0 - 0.363217 2024-04-16T08:41:45.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3171 16 1935 1919 0.0378182 0 - 0.363217 2024-04-16T08:41:46.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3172 16 1935 1919 0.0378063 0 - 0.363217 2024-04-16T08:41:47.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3173 16 1935 1919 0.0377944 0 - 0.363217 2024-04-16T08:41:48.289 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3174 16 1935 1919 0.0377825 0 - 0.363217 2024-04-16T08:41:49.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3175 16 1935 1919 0.0377706 0 - 0.363217 2024-04-16T08:41:50.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3176 16 1935 1919 0.0377587 0 - 0.363217 2024-04-16T08:41:51.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3177 16 1935 1919 0.0377468 0 - 0.363217 2024-04-16T08:41:52.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3178 16 1935 1919 0.0377349 0 - 0.363217 2024-04-16T08:41:53.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3179 16 1935 1919 0.0377231 0 - 0.363217 2024-04-16T08:41:54.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:41:54.292283+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:41:54.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:41:54.291 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3180 16 1935 1919 0.0377112 0 - 0.363217 2024-04-16T08:41:55.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3181 16 1935 1919 0.0376993 0 - 0.363217 2024-04-16T08:41:56.290 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3182 16 1935 1919 0.0376875 0 - 0.363217 2024-04-16T08:41:57.291 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3183 16 1935 1919 0.0376757 0 - 0.363217 2024-04-16T08:41:58.291 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3184 16 1935 1919 0.0376638 0 - 0.363217 2024-04-16T08:41:59.291 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3185 16 1935 1919 0.037652 0 - 0.363217 2024-04-16T08:42:00.291 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3186 16 1935 1919 0.0376402 0 - 0.363217 2024-04-16T08:42:01.291 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3187 16 1935 1919 0.0376284 0 - 0.363217 2024-04-16T08:42:02.291 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3188 16 1935 1919 0.0376166 0 - 0.363217 2024-04-16T08:42:03.291 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3189 16 1935 1919 0.0376048 0 - 0.363217 2024-04-16T08:42:04.292 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3190 16 1935 1919 0.037593 0 - 0.363217 2024-04-16T08:42:05.292 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3191 16 1935 1919 0.0375812 0 - 0.363217 2024-04-16T08:42:06.292 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3192 16 1935 1919 0.0375694 0 - 0.363217 2024-04-16T08:42:07.292 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3193 16 1935 1919 0.0375577 0 - 0.363217 2024-04-16T08:42:08.292 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3194 16 1935 1919 0.0375459 0 - 0.363217 2024-04-16T08:42:09.292 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3195 16 1935 1919 0.0375342 0 - 0.363217 2024-04-16T08:42:10.292 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3196 16 1935 1919 0.0375224 0 - 0.363217 2024-04-16T08:42:11.292 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3197 16 1935 1919 0.0375107 0 - 0.363217 2024-04-16T08:42:12.293 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3198 16 1935 1919 0.0374989 0 - 0.363217 2024-04-16T08:42:13.293 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3199 16 1935 1919 0.0374872 0 - 0.363217 2024-04-16T08:42:14.293 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:42:14.294907+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:42:14.293 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:42:14.293 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3200 16 1935 1919 0.0374755 0 - 0.363217 2024-04-16T08:42:15.293 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3201 16 1935 1919 0.0374638 0 - 0.363217 2024-04-16T08:42:16.293 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3202 16 1935 1919 0.0374521 0 - 0.363217 2024-04-16T08:42:17.293 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3203 16 1935 1919 0.0374404 0 - 0.363217 2024-04-16T08:42:18.293 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3204 16 1935 1919 0.0374287 0 - 0.363217 2024-04-16T08:42:19.294 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3205 16 1935 1919 0.037417 0 - 0.363217 2024-04-16T08:42:20.294 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3206 16 1935 1919 0.0374054 0 - 0.363217 2024-04-16T08:42:21.294 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3207 16 1935 1919 0.0373937 0 - 0.363217 2024-04-16T08:42:22.294 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3208 16 1935 1919 0.0373821 0 - 0.363217 2024-04-16T08:42:23.294 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3209 16 1935 1919 0.0373704 0 - 0.363217 2024-04-16T08:42:24.294 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3210 16 1935 1919 0.0373588 0 - 0.363217 2024-04-16T08:42:25.294 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3211 16 1935 1919 0.0373471 0 - 0.363217 2024-04-16T08:42:26.294 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3212 16 1935 1919 0.0373355 0 - 0.363217 2024-04-16T08:42:27.295 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3213 16 1935 1919 0.0373239 0 - 0.363217 2024-04-16T08:42:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:42:27.826 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-16T08:42:27.838 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T08:42:27.847 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T08:42:27.848 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T08:42:27.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-16T08:42:27.973+0000 7f5d5fe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T08:42:28.295 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3214 16 1935 1919 0.0373123 0 - 0.363217 2024-04-16T08:42:29.295 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3215 16 1935 1919 0.0373007 0 - 0.363217 2024-04-16T08:42:30.295 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3216 16 1935 1919 0.0372891 0 - 0.363217 2024-04-16T08:42:31.295 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3217 16 1935 1919 0.0372775 0 - 0.363217 2024-04-16T08:42:32.295 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3218 16 1935 1919 0.0372659 0 - 0.363217 2024-04-16T08:42:33.295 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3219 16 1935 1919 0.0372543 0 - 0.363217 2024-04-16T08:42:34.295 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:42:34.297527+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3220 16 1935 1919 0.0372427 0 - 0.363217 2024-04-16T08:42:35.296 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3221 16 1935 1919 0.0372312 0 - 0.363217 2024-04-16T08:42:36.296 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3222 16 1935 1919 0.0372196 0 - 0.363217 2024-04-16T08:42:37.296 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3223 16 1935 1919 0.0372081 0 - 0.363217 2024-04-16T08:42:38.296 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3224 16 1935 1919 0.0371965 0 - 0.363217 2024-04-16T08:42:39.296 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3225 16 1935 1919 0.037185 0 - 0.363217 2024-04-16T08:42:40.296 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3226 16 1935 1919 0.0371735 0 - 0.363217 2024-04-16T08:42:41.296 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3227 16 1935 1919 0.037162 0 - 0.363217 2024-04-16T08:42:42.297 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3228 16 1935 1919 0.0371504 0 - 0.363217 2024-04-16T08:42:43.297 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3229 16 1935 1919 0.0371389 0 - 0.363217 2024-04-16T08:42:44.297 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3230 16 1935 1919 0.0371274 0 - 0.363217 2024-04-16T08:42:45.297 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3231 16 1935 1919 0.0371159 0 - 0.363217 2024-04-16T08:42:46.297 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3232 16 1935 1919 0.0371045 0 - 0.363217 2024-04-16T08:42:47.297 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3233 16 1935 1919 0.037093 0 - 0.363217 2024-04-16T08:42:48.297 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3234 16 1935 1919 0.0370815 0 - 0.363217 2024-04-16T08:42:49.297 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3235 16 1935 1919 0.0370701 0 - 0.363217 2024-04-16T08:42:50.298 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3236 16 1935 1919 0.0370586 0 - 0.363217 2024-04-16T08:42:51.298 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3237 16 1935 1919 0.0370471 0 - 0.363217 2024-04-16T08:42:52.298 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3238 16 1935 1919 0.0370357 0 - 0.363217 2024-04-16T08:42:53.298 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3239 16 1935 1919 0.0370243 0 - 0.363217 2024-04-16T08:42:54.298 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:42:54.300235+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:42:54.299 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:42:54.299 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3240 16 1935 1919 0.0370128 0 - 0.363217 2024-04-16T08:42:55.298 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3241 16 1935 1919 0.0370014 0 - 0.363217 2024-04-16T08:42:56.298 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3242 16 1935 1919 0.03699 0 - 0.363217 2024-04-16T08:42:57.298 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3243 16 1935 1919 0.0369786 0 - 0.363217 2024-04-16T08:42:58.299 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3244 16 1935 1919 0.0369672 0 - 0.363217 2024-04-16T08:42:59.299 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3245 16 1935 1919 0.0369558 0 - 0.363217 2024-04-16T08:43:00.299 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3246 16 1935 1919 0.0369444 0 - 0.363217 2024-04-16T08:43:01.299 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3247 16 1935 1919 0.0369331 0 - 0.363217 2024-04-16T08:43:02.299 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3248 16 1935 1919 0.0369217 0 - 0.363217 2024-04-16T08:43:03.299 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3249 16 1935 1919 0.0369103 0 - 0.363217 2024-04-16T08:43:04.299 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3250 16 1935 1919 0.036899 0 - 0.363217 2024-04-16T08:43:05.300 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3251 16 1935 1919 0.0368876 0 - 0.363217 2024-04-16T08:43:06.300 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3252 16 1935 1919 0.0368763 0 - 0.363217 2024-04-16T08:43:07.300 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3253 16 1935 1919 0.0368649 0 - 0.363217 2024-04-16T08:43:08.300 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3254 16 1935 1919 0.0368536 0 - 0.363217 2024-04-16T08:43:09.300 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3255 16 1935 1919 0.0368423 0 - 0.363217 2024-04-16T08:43:10.300 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3256 16 1935 1919 0.036831 0 - 0.363217 2024-04-16T08:43:11.300 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3257 16 1935 1919 0.0368197 0 - 0.363217 2024-04-16T08:43:12.300 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3258 16 1935 1919 0.0368084 0 - 0.363217 2024-04-16T08:43:13.301 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3259 16 1935 1919 0.0367971 0 - 0.363217 2024-04-16T08:43:14.301 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:43:14.302874+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:43:14.301 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:43:14.301 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3260 16 1935 1919 0.0367858 0 - 0.363217 2024-04-16T08:43:15.301 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3261 16 1935 1919 0.0367745 0 - 0.363217 2024-04-16T08:43:16.301 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3262 16 1935 1919 0.0367632 0 - 0.363217 2024-04-16T08:43:17.301 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3263 16 1935 1919 0.036752 0 - 0.363217 2024-04-16T08:43:18.301 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3264 16 1935 1919 0.0367407 0 - 0.363217 2024-04-16T08:43:19.301 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3265 16 1935 1919 0.0367294 0 - 0.363217 2024-04-16T08:43:20.302 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3266 16 1935 1919 0.0367182 0 - 0.363217 2024-04-16T08:43:21.302 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3267 16 1935 1919 0.036707 0 - 0.363217 2024-04-16T08:43:22.302 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3268 16 1935 1919 0.0366957 0 - 0.363217 2024-04-16T08:43:23.302 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3269 16 1935 1919 0.0366845 0 - 0.363217 2024-04-16T08:43:24.302 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3270 16 1935 1919 0.0366733 0 - 0.363217 2024-04-16T08:43:25.302 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3271 16 1935 1919 0.0366621 0 - 0.363217 2024-04-16T08:43:26.302 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3272 16 1935 1919 0.0366509 0 - 0.363217 2024-04-16T08:43:27.302 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3273 16 1935 1919 0.0366397 0 - 0.363217 2024-04-16T08:43:28.303 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3274 16 1935 1919 0.0366285 0 - 0.363217 2024-04-16T08:43:29.303 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3275 16 1935 1919 0.0366173 0 - 0.363217 2024-04-16T08:43:30.303 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3276 16 1935 1919 0.0366061 0 - 0.363217 2024-04-16T08:43:31.303 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3277 16 1935 1919 0.0365949 0 - 0.363217 2024-04-16T08:43:32.303 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3278 16 1935 1919 0.0365838 0 - 0.363217 2024-04-16T08:43:33.303 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3279 16 1935 1919 0.0365726 0 - 0.363217 2024-04-16T08:43:34.303 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:43:34.305569+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:43:34.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:43:34.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3280 16 1935 1919 0.0365615 0 - 0.363217 2024-04-16T08:43:35.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3281 16 1935 1919 0.0365503 0 - 0.363217 2024-04-16T08:43:36.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3282 16 1935 1919 0.0365392 0 - 0.363217 2024-04-16T08:43:37.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3283 16 1935 1919 0.0365281 0 - 0.363217 2024-04-16T08:43:38.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3284 16 1935 1919 0.0365169 0 - 0.363217 2024-04-16T08:43:39.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3285 16 1935 1919 0.0365058 0 - 0.363217 2024-04-16T08:43:40.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3286 16 1935 1919 0.0364947 0 - 0.363217 2024-04-16T08:43:41.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3287 16 1935 1919 0.0364836 0 - 0.363217 2024-04-16T08:43:42.304 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3288 16 1935 1919 0.0364725 0 - 0.363217 2024-04-16T08:43:43.305 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3289 16 1935 1919 0.0364614 0 - 0.363217 2024-04-16T08:43:44.305 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3290 16 1935 1919 0.0364503 0 - 0.363217 2024-04-16T08:43:45.305 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3291 16 1935 1919 0.0364393 0 - 0.363217 2024-04-16T08:43:46.305 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3292 16 1935 1919 0.0364282 0 - 0.363217 2024-04-16T08:43:47.305 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3293 16 1935 1919 0.0364171 0 - 0.363217 2024-04-16T08:43:48.305 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3294 16 1935 1919 0.0364061 0 - 0.363217 2024-04-16T08:43:49.305 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3295 16 1935 1919 0.036395 0 - 0.363217 2024-04-16T08:43:50.306 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3296 16 1935 1919 0.036384 0 - 0.363217 2024-04-16T08:43:51.306 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3297 16 1935 1919 0.0363729 0 - 0.363217 2024-04-16T08:43:52.306 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3298 16 1935 1919 0.0363619 0 - 0.363217 2024-04-16T08:43:53.306 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3299 16 1935 1919 0.0363509 0 - 0.363217 2024-04-16T08:43:54.306 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:43:54.308310+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:43:54.307 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:43:54.307 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3300 16 1935 1919 0.0363399 0 - 0.363217 2024-04-16T08:43:55.306 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3301 16 1935 1919 0.0363289 0 - 0.363217 2024-04-16T08:43:56.306 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3302 16 1935 1919 0.0363179 0 - 0.363217 2024-04-16T08:43:57.306 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3303 16 1935 1919 0.0363069 0 - 0.363217 2024-04-16T08:43:58.307 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3304 16 1935 1919 0.0362959 0 - 0.363217 2024-04-16T08:43:59.307 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3305 16 1935 1919 0.0362849 0 - 0.363217 2024-04-16T08:44:00.307 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3306 16 1935 1919 0.0362739 0 - 0.363217 2024-04-16T08:44:01.307 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3307 16 1935 1919 0.036263 0 - 0.363217 2024-04-16T08:44:02.307 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3308 16 1935 1919 0.036252 0 - 0.363217 2024-04-16T08:44:03.307 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3309 16 1935 1919 0.036241 0 - 0.363217 2024-04-16T08:44:04.307 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3310 16 1935 1919 0.0362301 0 - 0.363217 2024-04-16T08:44:05.308 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3311 16 1935 1919 0.0362192 0 - 0.363217 2024-04-16T08:44:06.308 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3312 16 1935 1919 0.0362082 0 - 0.363217 2024-04-16T08:44:07.308 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3313 16 1935 1919 0.0361973 0 - 0.363217 2024-04-16T08:44:08.308 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3314 16 1935 1919 0.0361864 0 - 0.363217 2024-04-16T08:44:09.308 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3315 16 1935 1919 0.0361754 0 - 0.363217 2024-04-16T08:44:10.308 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3316 16 1935 1919 0.0361645 0 - 0.363217 2024-04-16T08:44:11.308 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3317 16 1935 1919 0.0361536 0 - 0.363217 2024-04-16T08:44:12.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3318 16 1935 1919 0.0361427 0 - 0.363217 2024-04-16T08:44:13.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3319 16 1935 1919 0.0361319 0 - 0.363217 2024-04-16T08:44:14.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:44:14.310948+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:44:14.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:44:14.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3320 16 1935 1919 0.036121 0 - 0.363217 2024-04-16T08:44:15.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3321 16 1935 1919 0.0361101 0 - 0.363217 2024-04-16T08:44:16.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3322 16 1935 1919 0.0360992 0 - 0.363217 2024-04-16T08:44:17.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3323 16 1935 1919 0.0360884 0 - 0.363217 2024-04-16T08:44:18.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3324 16 1935 1919 0.0360775 0 - 0.363217 2024-04-16T08:44:19.309 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3325 16 1935 1919 0.0360667 0 - 0.363217 2024-04-16T08:44:20.310 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3326 16 1935 1919 0.0360558 0 - 0.363217 2024-04-16T08:44:21.310 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3327 16 1935 1919 0.036045 0 - 0.363217 2024-04-16T08:44:22.311 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3328 16 1935 1919 0.0360341 0 - 0.363217 2024-04-16T08:44:23.310 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3329 16 1935 1919 0.0360233 0 - 0.363217 2024-04-16T08:44:24.310 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3330 16 1935 1919 0.0360125 0 - 0.363217 2024-04-16T08:44:25.310 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3331 16 1935 1919 0.0360017 0 - 0.363217 2024-04-16T08:44:26.311 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3332 16 1935 1919 0.0359909 0 - 0.363217 2024-04-16T08:44:27.311 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3333 16 1935 1919 0.0359801 0 - 0.363217 2024-04-16T08:44:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T08:44:27.892 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T08:44:27.912 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T08:44:27.913 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T08:44:27.913 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T08:44:27.913 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T08:44:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T08:44:27.917 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2024-04-16T08:44:27.934 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T08:44:27.935 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T08:44:27.935 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T08:44:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T08:44:27.935 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-04-16T08:44:27.953 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T08:44:27.953 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T08:44:27.953 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T08:44:28.024 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T08:44:28.024 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-16T08:44:28.024 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T08:44:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T08:44:28.025 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log' 2024-04-16T08:44:28.042 INFO:tasks.ceph.osd.3:Stopped 2024-04-16T08:44:28.042 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T08:44:28.043 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-16T08:44:28.043 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T08:44:28.046 INFO:tasks.ceph.mgr.x:Stopped 2024-04-16T08:44:28.046 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T08:44:28.047 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-16T08:44:28.047 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T08:44:28.047 INFO:tasks.ceph.mon.a:Stopped 2024-04-16T08:44:28.047 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T08:44:28.048 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-16T08:44:28.132 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-16T08:44:28.133 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-16T08:44:28.202 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-16T08:44:28.202 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-16T08:44:28.277 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-16T08:44:28.277 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-16T08:44:28.311 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3334 16 1935 1919 0.0359693 0 - 0.363217 2024-04-16T08:44:28.402 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-16T08:44:28.403 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-16T08:44:28.497 INFO:tasks.ceph:Archiving mon data... 2024-04-16T08:44:28.498 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962/data/mon.a.tgz 2024-04-16T08:44:28.499 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-04-16T08:44:28.516 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.uOPsf4X259 2024-04-16T08:44:28.517 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.uOPsf4X259 2024-04-16T08:44:28.675 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.uOPsf4X259 2024-04-16T08:44:28.763 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.uOPsf4X259 is 210KB 2024-04-16T08:44:28.857 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.uOPsf4X259 2024-04-16T08:44:28.872 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T08:44:28.872 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T08:44:28.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T08:44:28.996 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T08:44:29.014 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T08:44:29.014 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T08:44:29.016 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962/remote/smithi079/crash 2024-04-16T08:44:29.017 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T08:44:29.064 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962/remote/smithi167/crash 2024-04-16T08:44:29.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T08:44:29.114 INFO:tasks.ceph:Compressing logs... 2024-04-16T08:44:29.114 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T08:44:29.116 DEBUG:teuthology.orchestra.run.smithi167:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T08:44:29.152 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29354.log 2024-04-16T08:44:29.152 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29274.log 2024-04-16T08:44:29.152 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-16T08:44:29.153 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.29354.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29354.log.gz 2024-04-16T08:44:29.153 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.29274.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-16T08:44:29.153 INFO:teuthology.orchestra.run.smithi167.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29274.log.gz 2024-04-16T08:44:29.153 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29234.log 2024-04-16T08:44:29.153 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-16T08:44:29.153 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.29234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29314.log 2024-04-16T08:44:29.153 INFO:teuthology.orchestra.run.smithi167.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29234.log.gz 2024-04-16T08:44:29.153 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.29314.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29314.log.gz 2024-04-16T08:44:29.222 INFO:teuthology.orchestra.run.smithi167.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-16T08:44:29.275 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30450.log 2024-04-16T08:44:29.275 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40504.log 2024-04-16T08:44:29.275 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41848.log 2024-04-16T08:44:29.276 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.30450.log.gz 2024-04-16T08:44:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31038.log 2024-04-16T08:44:29.276 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40504.log.gz 2024-04-16T08:44:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37648.log 2024-04-16T08:44:29.276 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41848.log.gz 2024-04-16T08:44:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43948.log 2024-04-16T08:44:29.276 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31038.log.gz 2024-04-16T08:44:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37648.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-16T08:44:29.276 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37648.log.gz 2024-04-16T08:44:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45881.log 2024-04-16T08:44:29.277 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43948.log.gz 2024-04-16T08:44:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44368.log 2024-04-16T08:44:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32585.log 2024-04-16T08:44:29.277 INFO:teuthology.orchestra.run.smithi079.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45881.log.gz 2024-04-16T08:44:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38782.log 2024-04-16T08:44:29.277 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44368.log.gz 2024-04-16T08:44:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43654.log 2024-04-16T08:44:29.277 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32585.log.gz 2024-04-16T08:44:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30274.log 2024-04-16T08:44:29.278 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38782.log.gz 2024-04-16T08:44:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42352.log 2024-04-16T08:44:29.278 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43654.log.gz 2024-04-16T08:44:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31220.log 2024-04-16T08:44:29.278 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30274.log.gz 2024-04-16T08:44:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40084.log 2024-04-16T08:44:29.278 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42352.log.gz 2024-04-16T08:44:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33728.log 2024-04-16T08:44:29.279 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31220.log.gz 2024-04-16T08:44:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42898.log 2024-04-16T08:44:29.279 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40084.log.gz 2024-04-16T08:44:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38152.log 2024-04-16T08:44:29.279 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33728.log.gz 2024-04-16T08:44:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43360.log 2024-04-16T08:44:29.279 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42898.log.gz 2024-04-16T08:44:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32476.log 2024-04-16T08:44:29.280 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38152.log.gz 2024-04-16T08:44:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36202.log 2024-04-16T08:44:29.280 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43360.log.gz 2024-04-16T08:44:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38698.log 2024-04-16T08:44:29.280 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32476.log.gz 2024-04-16T08:44:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38740.log 2024-04-16T08:44:29.280 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz 2024-04-16T08:44:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32327.log 2024-04-16T08:44:29.280 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38698.log.gz 2024-04-16T08:44:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40756.log 2024-04-16T08:44:29.281 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38740.log.gz 2024-04-16T08:44:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39790.log 2024-04-16T08:44:29.281 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32327.log.gz 2024-04-16T08:44:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34804.log 2024-04-16T08:44:29.281 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40756.log.gz 2024-04-16T08:44:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42058.log 2024-04-16T08:44:29.281 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39790.log.gz 2024-04-16T08:44:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30551.log 2024-04-16T08:44:29.282 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34804.log.gz 2024-04-16T08:44:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40546.log 2024-04-16T08:44:29.282 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42058.log.gz 2024-04-16T08:44:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35280.log 2024-04-16T08:44:29.282 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30551.log.gz 2024-04-16T08:44:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38362.log 2024-04-16T08:44:29.282 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.40546.log.gz 2024-04-16T08:44:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41428.log 2024-04-16T08:44:29.283 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35280.log.gz 2024-04-16T08:44:29.283 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32376.log 2024-04-16T08:44:29.283 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38362.log.gz 2024-04-16T08:44:29.283 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35644.log 2024-04-16T08:44:29.283 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41428.log.gz 2024-04-16T08:44:29.283 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46004.log 2024-04-16T08:44:29.283 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32376.log.gz 2024-04-16T08:44:29.283 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37606.log 2024-04-16T08:44:29.283 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35644.log.gz 2024-04-16T08:44:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-04-16T08:44:29.284 INFO:teuthology.orchestra.run.smithi079.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.46004.log.gz 2024-04-16T08:44:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39160.log 2024-04-16T08:44:29.284 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37606.log.gz 2024-04-16T08:44:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43486.log 2024-04-16T08:44:29.284 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-04-16T08:44:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45310.log 2024-04-16T08:44:29.284 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39160.log.gz 2024-04-16T08:44:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37435.log 2024-04-16T08:44:29.285 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43486.log.gz 2024-04-16T08:44:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45602.log 2024-04-16T08:44:29.285 INFO:teuthology.orchestra.run.smithi079.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45310.log.gz 2024-04-16T08:44:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45447.log 2024-04-16T08:44:29.285 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37435.log.gz 2024-04-16T08:44:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30719.log 2024-04-16T08:44:29.285 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45602.log.gz 2024-04-16T08:44:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37561.log 2024-04-16T08:44:29.286 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45447.log.gz 2024-04-16T08:44:29.286 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35379.log 2024-04-16T08:44:29.286 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30719.log.gz 2024-04-16T08:44:29.286 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44284.log 2024-04-16T08:44:29.286 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37561.log.gz 2024-04-16T08:44:29.286 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-04-16T08:44:29.286 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35379.log.gz 2024-04-16T08:44:29.286 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41134.log 2024-04-16T08:44:29.286 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44284.log.gz 2024-04-16T08:44:29.287 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40042.log 2024-04-16T08:44:29.287 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-04-16T08:44:29.287 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45942.log 2024-04-16T08:44:29.287 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41134.log.gz 2024-04-16T08:44:29.287 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39664.log 2024-04-16T08:44:29.287 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40042.log.gz 2024-04-16T08:44:29.287 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44032.log 2024-04-16T08:44:29.287 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45942.log.gz 2024-04-16T08:44:29.288 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43276.log 2024-04-16T08:44:29.288 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39664.log.gz 2024-04-16T08:44:29.288 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30550.log 2024-04-16T08:44:29.288 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44032.log.gz 2024-04-16T08:44:29.288 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41638.log 2024-04-16T08:44:29.288 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43276.log.gz 2024-04-16T08:44:29.288 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30912.log 2024-04-16T08:44:29.288 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30550.log.gz 2024-04-16T08:44:29.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45633.log 2024-04-16T08:44:29.289 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41638.log.gz 2024-04-16T08:44:29.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44830.log 2024-04-16T08:44:29.289 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30912.log.gz 2024-04-16T08:44:29.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31579.log 2024-04-16T08:44:29.289 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45633.log.gz 2024-04-16T08:44:29.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40588.log 2024-04-16T08:44:29.289 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.44830.log.gz 2024-04-16T08:44:29.289 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36709.log 2024-04-16T08:44:29.290 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31579.log.gz 2024-04-16T08:44:29.290 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43402.log 2024-04-16T08:44:29.290 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40588.log.gz 2024-04-16T08:44:29.290 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31628.log 2024-04-16T08:44:29.290 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36709.log.gz 2024-04-16T08:44:29.290 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36287.log 2024-04-16T08:44:29.290 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43402.log.gz 2024-04-16T08:44:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39034.log 2024-04-16T08:44:29.291 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31628.log.gz 2024-04-16T08:44:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37393.log 2024-04-16T08:44:29.291 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36287.log.gz 2024-04-16T08:44:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31830.log 2024-04-16T08:44:29.291 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39034.log.gz 2024-04-16T08:44:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37256.log 2024-04-16T08:44:29.291 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37393.log.gz 2024-04-16T08:44:29.292 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33477.log 2024-04-16T08:44:29.292 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31830.log.gz 2024-04-16T08:44:29.292 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37942.log 2024-04-16T08:44:29.292 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37256.log.gz 2024-04-16T08:44:29.292 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30358.log 2024-04-16T08:44:29.292 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33477.log.gz 2024-04-16T08:44:29.292 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37003.log 2024-04-16T08:44:29.292 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37942.log.gz 2024-04-16T08:44:29.293 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36583.log 2024-04-16T08:44:29.293 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30358.log.gz 2024-04-16T08:44:29.293 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41932.log 2024-04-16T08:44:29.293 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37003.log.gz 2024-04-16T08:44:29.293 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44788.log 2024-04-16T08:44:29.293 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36583.log.gz 2024-04-16T08:44:29.293 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42814.log 2024-04-16T08:44:29.293 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41932.log.gz 2024-04-16T08:44:29.294 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39370.log 2024-04-16T08:44:29.294 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44788.log.gz 2024-04-16T08:44:29.294 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44116.log 2024-04-16T08:44:29.294 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42814.log.gz 2024-04-16T08:44:29.294 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33824.log 2024-04-16T08:44:29.294 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39370.log.gz 2024-04-16T08:44:29.294 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37858.log 2024-04-16T08:44:29.294 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44116.log.gz 2024-04-16T08:44:29.294 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42520.log 2024-04-16T08:44:29.295 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33824.log.gz 2024-04-16T08:44:29.295 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45082.log 2024-04-16T08:44:29.295 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37858.log.gz 2024-04-16T08:44:29.295 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30870.log 2024-04-16T08:44:29.295 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42520.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42520.log.gz 2024-04-16T08:44:29.295 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34855.log 2024-04-16T08:44:29.295 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45082.log.gz 2024-04-16T08:44:29.295 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30549.log 2024-04-16T08:44:29.296 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30870.log.gz 2024-04-16T08:44:29.296 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43864.log 2024-04-16T08:44:29.296 INFO:teuthology.orchestra.run.smithi079.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.34855.log.gz 2024-04-16T08:44:29.296 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44998.log 2024-04-16T08:44:29.296 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30549.log.gz 2024-04-16T08:44:29.296 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32539.log 2024-04-16T08:44:29.296 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43864.log.gz 2024-04-16T08:44:29.296 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36034.log 2024-04-16T08:44:29.297 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44998.log.gz 2024-04-16T08:44:29.297 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38236.log 2024-04-16T08:44:29.297 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32539.log.gz 2024-04-16T08:44:29.297 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40420.log 2024-04-16T08:44:29.297 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36034.log.gz 2024-04-16T08:44:29.297 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 2024-04-16T08:44:29.297 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38236.log.gz 2024-04-16T08:44:29.297 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37087.log 2024-04-16T08:44:29.297 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40420.log.gz 2024-04-16T08:44:29.298 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41764.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz --verbose 2024-04-16T08:44:29.298 INFO:teuthology.orchestra.run.smithi079.stderr: -- /var/log/ceph/ceph-client.admin.45250.log 2024-04-16T08:44:29.298 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42394.log 2024-04-16T08:44:29.298 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37087.log.gz 2024-04-16T08:44:29.298 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40000.log 2024-04-16T08:44:29.298 INFO:teuthology.orchestra.run.smithi079.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45250.log.gz 2024-04-16T08:44:29.298 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41092.log 2024-04-16T08:44:29.298 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42394.log.gz 2024-04-16T08:44:29.299 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2024-04-16T08:44:29.299 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40000.log.gz 2024-04-16T08:44:29.299 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39580.log 2024-04-16T08:44:29.299 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41092.log.gz 2024-04-16T08:44:29.299 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35950.log 2024-04-16T08:44:29.299 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2024-04-16T08:44:29.299 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44578.log 2024-04-16T08:44:29.299 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39580.log.gz 2024-04-16T08:44:29.299 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37045.log 2024-04-16T08:44:29.300 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35950.log.gz 2024-04-16T08:44:29.300 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42478.log 2024-04-16T08:44:29.300 INFO:teuthology.orchestra.run.smithi079.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.44578.log.gz 2024-04-16T08:44:29.300 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33137.log 2024-04-16T08:44:29.300 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37045.log.gz 2024-04-16T08:44:29.300 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42982.log 2024-04-16T08:44:29.300 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42478.log.gz 2024-04-16T08:44:29.300 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32416.log 2024-04-16T08:44:29.300 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33137.log.gz 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31929.log 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42982.log.gz 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32416.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32416.log.gz 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39706.log 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31929.log.gz 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38488.log 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39706.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39706.log.gz 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43990.log 2024-04-16T08:44:29.301 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43696.log 2024-04-16T08:44:29.302 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38488.log.gz 2024-04-16T08:44:29.302 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43990.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43990.log.gz 2024-04-16T08:44:29.302 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37213.log 2024-04-16T08:44:29.302 INFO:teuthology.orchestra.run.smithi079.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T08:44:29.302 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31978.log 2024-04-16T08:44:29.302 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43696.log.gz 2024-04-16T08:44:29.302 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43906.log 2024-04-16T08:44:29.302 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37213.log.gz 2024-04-16T08:44:29.302 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35740.log 2024-04-16T08:44:29.303 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31978.log.gz 2024-04-16T08:44:29.303 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38320.log 2024-04-16T08:44:29.303 INFO:teuthology.orchestra.run.smithi079.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.43906.log.gz 2024-04-16T08:44:29.303 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32720.log 2024-04-16T08:44:29.303 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35740.log.gz 2024-04-16T08:44:29.303 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37519.log 2024-04-16T08:44:29.303 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38320.log.gz 2024-04-16T08:44:29.303 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35992.log 2024-04-16T08:44:29.303 INFO:teuthology.orchestra.run.smithi079.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.32720.log.gz 2024-04-16T08:44:29.304 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39118.log 2024-04-16T08:44:29.304 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37519.log.gz 2024-04-16T08:44:29.304 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38908.log 2024-04-16T08:44:29.304 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.35992.log.gz 2024-04-16T08:44:29.304 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32198.log 2024-04-16T08:44:29.304 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39118.log.gz 2024-04-16T08:44:29.304 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43318.log 2024-04-16T08:44:29.304 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38908.log.gz 2024-04-16T08:44:29.304 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43234.log 2024-04-16T08:44:29.305 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32198.log.gz 2024-04-16T08:44:29.305 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29994.log 2024-04-16T08:44:29.305 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43318.log.gz 2024-04-16T08:44:29.305 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45696.log 2024-04-16T08:44:29.305 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43234.log.gz 2024-04-16T08:44:29.305 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37308.log 2024-04-16T08:44:29.305 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29994.log.gz 2024-04-16T08:44:29.305 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34075.log 2024-04-16T08:44:29.305 INFO:teuthology.orchestra.run.smithi079.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45696.log.gz 2024-04-16T08:44:29.306 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30408.log 2024-04-16T08:44:29.306 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37308.log.gz 2024-04-16T08:44:29.306 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37690.log 2024-04-16T08:44:29.306 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34075.log.gz 2024-04-16T08:44:29.306 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38026.log 2024-04-16T08:44:29.306 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30408.log.gz 2024-04-16T08:44:29.306 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44956.log 2024-04-16T08:44:29.306 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37690.log.gz 2024-04-16T08:44:29.306 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39958.log 2024-04-16T08:44:29.307 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38026.log.gz 2024-04-16T08:44:29.307 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37129.log 2024-04-16T08:44:29.307 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44956.log.gz 2024-04-16T08:44:29.307 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37171.log 2024-04-16T08:44:29.307 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39958.log.gz 2024-04-16T08:44:29.307 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33428.log 2024-04-16T08:44:29.307 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37129.log.gz 2024-04-16T08:44:29.307 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37171.log: gzip -5 50.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.37171.log.gz -- 2024-04-16T08:44:29.307 INFO:teuthology.orchestra.run.smithi079.stderr: /var/log/ceph/ceph-client.admin.44662.log 2024-04-16T08:44:29.308 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35071.log 2024-04-16T08:44:29.308 INFO:teuthology.orchestra.run.smithi079.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.33428.log.gz 2024-04-16T08:44:29.308 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42856.log 2024-04-16T08:44:29.308 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44662.log.gz 2024-04-16T08:44:29.308 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45417.log 2024-04-16T08:44:29.308 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35071.log.gz 2024-04-16T08:44:29.308 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45974.log 2024-04-16T08:44:29.308 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42856.log.gz 2024-04-16T08:44:29.308 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37351.log 2024-04-16T08:44:29.309 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45417.log.gz 2024-04-16T08:44:29.309 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34444.log 2024-04-16T08:44:29.309 INFO:teuthology.orchestra.run.smithi079.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45974.log.gz 2024-04-16T08:44:29.309 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42604.log 2024-04-16T08:44:29.309 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37351.log.gz 2024-04-16T08:44:29.309 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43150.log 2024-04-16T08:44:29.309 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34444.log.gz 2024-04-16T08:44:29.309 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-04-16T08:44:29.310 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42604.log.gz 2024-04-16T08:44:29.310 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40924.log 2024-04-16T08:44:29.310 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43150.log.gz 2024-04-16T08:44:29.310 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39244.log 2024-04-16T08:44:29.310 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-04-16T08:44:29.310 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38992.log 2024-04-16T08:44:29.310 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40924.log.gz 2024-04-16T08:44:29.310 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40882.log 2024-04-16T08:44:29.311 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39244.log.gz 2024-04-16T08:44:29.311 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44494.log 2024-04-16T08:44:29.311 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38992.log.gz 2024-04-16T08:44:29.311 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40840.log 2024-04-16T08:44:29.311 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40882.log.gz 2024-04-16T08:44:29.311 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40714.log 2024-04-16T08:44:29.311 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44494.log.gz 2024-04-16T08:44:29.311 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36160.log 2024-04-16T08:44:29.312 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40840.log.gz 2024-04-16T08:44:29.312 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44914.log 2024-04-16T08:44:29.312 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40714.log.gz 2024-04-16T08:44:29.312 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36160.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-16T08:44:29.312 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36160.log.gz 2024-04-16T08:44:29.312 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45849.log 2024-04-16T08:44:29.312 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44914.log.gz 2024-04-16T08:44:29.312 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30828.log 2024-04-16T08:44:29.312 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30316.log 2024-04-16T08:44:29.313 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45849.log.gz 2024-04-16T08:44:29.313 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36118.log 2024-04-16T08:44:29.313 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30828.log.gz 2024-04-16T08:44:29.313 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34124.log 2024-04-16T08:44:29.313 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.30316.log.gz 2024-04-16T08:44:29.313 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36118.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.36118.log.gz 2024-04-16T08:44:29.313 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43822.log 2024-04-16T08:44:29.313 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36793.log 2024-04-16T08:44:29.313 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34124.log.gz 2024-04-16T08:44:29.314 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2024-04-16T08:44:29.314 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43822.log.gz 2024-04-16T08:44:29.314 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36793.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36793.log.gz 2024-04-16T08:44:29.314 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44872.log 2024-04-16T08:44:29.314 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40672.log 2024-04-16T08:44:29.314 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2024-04-16T08:44:29.314 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45819.log 2024-04-16T08:44:29.314 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44872.log.gz 2024-04-16T08:44:29.315 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40672.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40672.log.gz 2024-04-16T08:44:29.317 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43066.log 2024-04-16T08:44:29.317 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45040.log 2024-04-16T08:44:29.317 INFO:teuthology.orchestra.run.smithi079.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45819.log.gz 2024-04-16T08:44:29.317 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39452.log 2024-04-16T08:44:29.317 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43066.log.gz 2024-04-16T08:44:29.317 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38866.log 2024-04-16T08:44:29.317 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45040.log.gz 2024-04-16T08:44:29.318 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41218.log 2024-04-16T08:44:29.318 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39452.log.gz 2024-04-16T08:44:29.318 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32990.log 2024-04-16T08:44:29.318 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38866.log.gz 2024-04-16T08:44:29.318 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35330.log 2024-04-16T08:44:29.318 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41218.log.gz 2024-04-16T08:44:29.318 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34326.log 2024-04-16T08:44:29.318 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32990.log.gz 2024-04-16T08:44:29.319 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39328.log 2024-04-16T08:44:29.319 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35330.log.gz 2024-04-16T08:44:29.319 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34326.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34326.log.gz 2024-04-16T08:44:29.319 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42772.log 2024-04-16T08:44:29.319 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38446.log 2024-04-16T08:44:29.319 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39328.log.gz 2024-04-16T08:44:29.319 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39832.log 2024-04-16T08:44:29.319 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42772.log.gz 2024-04-16T08:44:29.319 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38446.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38446.log.gz 2024-04-16T08:44:29.320 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40462.log 2024-04-16T08:44:29.320 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39412.log 2024-04-16T08:44:29.320 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39832.log.gz 2024-04-16T08:44:29.320 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45786.log 2024-04-16T08:44:29.320 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40462.log.gz 2024-04-16T08:44:29.320 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31080.log 2024-04-16T08:44:29.320 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39412.log.gz 2024-04-16T08:44:29.320 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45786.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45786.log.gzgzip 2024-04-16T08:44:29.320 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38194.log 2024-04-16T08:44:29.321 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32893.log 2024-04-16T08:44:29.321 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31080.log.gz 2024-04-16T08:44:29.321 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33359.log 2024-04-16T08:44:29.321 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38194.log.gz 2024-04-16T08:44:29.321 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39538.log 2024-04-16T08:44:29.321 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32893.log.gz 2024-04-16T08:44:29.321 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41680.log 2024-04-16T08:44:29.321 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33359.log.gz 2024-04-16T08:44:29.321 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-04-16T08:44:29.322 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39538.log.gz 2024-04-16T08:44:29.322 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43612.log 2024-04-16T08:44:29.322 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41680.log.gz 2024-04-16T08:44:29.322 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38614.log 2024-04-16T08:44:29.322 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz 2024-04-16T08:44:29.322 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40210.log 2024-04-16T08:44:29.322 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43612.log.gz 2024-04-16T08:44:29.322 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44620.log 2024-04-16T08:44:29.323 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38614.log.gz 2024-04-16T08:44:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45477.log 2024-04-16T08:44:29.323 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40210.log.gz 2024-04-16T08:44:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35581.log 2024-04-16T08:44:29.323 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44620.log.gz 2024-04-16T08:44:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45280.log 2024-04-16T08:44:29.323 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45477.log.gz 2024-04-16T08:44:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39202.log 2024-04-16T08:44:29.324 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35581.log.gz 2024-04-16T08:44:29.324 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44746.log 2024-04-16T08:44:29.324 INFO:teuthology.orchestra.run.smithi079.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45280.log.gz 2024-04-16T08:44:29.324 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42688.log 2024-04-16T08:44:29.324 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2024-04-16T08:44:29.324 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31287.log 2024-04-16T08:44:29.324 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44746.log.gz 2024-04-16T08:44:29.324 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30954.log 2024-04-16T08:44:29.324 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42688.log.gz 2024-04-16T08:44:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45726.log 2024-04-16T08:44:29.325 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31287.log.gz 2024-04-16T08:44:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36625.log 2024-04-16T08:44:29.325 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30954.log.gz 2024-04-16T08:44:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32238.log 2024-04-16T08:44:29.325 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45726.log.gz 2024-04-16T08:44:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41050.log 2024-04-16T08:44:29.325 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36625.log.gz 2024-04-16T08:44:29.326 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35908.log 2024-04-16T08:44:29.326 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32238.log.gz 2024-04-16T08:44:29.326 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38068.log 2024-04-16T08:44:29.326 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41050.log.gz 2024-04-16T08:44:29.326 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43108.log 2024-04-16T08:44:29.326 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35908.log.gz 2024-04-16T08:44:29.326 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38824.log 2024-04-16T08:44:29.326 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38068.log.gz 2024-04-16T08:44:29.326 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43780.log 2024-04-16T08:44:29.327 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43108.log.gz 2024-04-16T08:44:29.327 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43192.log 2024-04-16T08:44:29.327 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38824.log.gz 2024-04-16T08:44:29.327 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42436.log 2024-04-16T08:44:29.327 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43780.log.gz 2024-04-16T08:44:29.327 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32069.log 2024-04-16T08:44:29.327 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43192.log.gz 2024-04-16T08:44:29.327 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35240.log 2024-04-16T08:44:29.327 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42436.log.gz 2024-04-16T08:44:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36961.log 2024-04-16T08:44:29.328 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32069.log.gz 2024-04-16T08:44:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41722.log 2024-04-16T08:44:29.328 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35240.log.gz 2024-04-16T08:44:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36330.log 2024-04-16T08:44:29.328 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36961.log.gz 2024-04-16T08:44:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29810.log 2024-04-16T08:44:29.328 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41722.log.gz 2024-04-16T08:44:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42100.log 2024-04-16T08:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36330.log.gz 2024-04-16T08:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41260.log 2024-04-16T08:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29810.log.gz 2024-04-16T08:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36877.log 2024-04-16T08:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42100.log.gz 2024-04-16T08:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39076.log 2024-04-16T08:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41260.log.gz 2024-04-16T08:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30996.log 2024-04-16T08:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36877.log.gz 2024-04-16T08:44:29.330 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35782.log 2024-04-16T08:44:29.330 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39076.log.gz 2024-04-16T08:44:29.330 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41344.log 2024-04-16T08:44:29.330 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30996.log.gz 2024-04-16T08:44:29.330 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31310.log 2024-04-16T08:44:29.330 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35782.log.gz 2024-04-16T08:44:29.330 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37774.log 2024-04-16T08:44:29.330 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41344.log.gz 2024-04-16T08:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.0.31310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43444.log 2024-04-16T08:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37774.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37774.log.gz 2024-04-16T08:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37732.log 2024-04-16T08:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-04-16T08:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43444.log.gz 2024-04-16T08:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41806.log 2024-04-16T08:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37732.log.gz 2024-04-16T08:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30218.log 2024-04-16T08:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-04-16T08:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41512.log 2024-04-16T08:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41806.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41806.log.gz 2024-04-16T08:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32843.log 2024-04-16T08:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30218.log.gz 2024-04-16T08:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36372.log 2024-04-16T08:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41512.log.gz 2024-04-16T08:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36835.log 2024-04-16T08:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32843.log.gz 2024-04-16T08:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36372.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36372.log.gz 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43570.log 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30720.log 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36835.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36835.log.gz 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45166.log 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43570.log.gz 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40630.log 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30720.log.gz 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40378.log 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45166.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45166.log.gz 2024-04-16T08:44:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33526.log 2024-04-16T08:44:29.334 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40630.log.gz 2024-04-16T08:44:29.334 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32158.log 2024-04-16T08:44:29.334 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40378.log.gz 2024-04-16T08:44:29.334 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45572.log 2024-04-16T08:44:29.334 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33526.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33526.log.gz 2024-04-16T08:44:29.334 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37984.log 2024-04-16T08:44:29.334 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32158.log.gz 2024-04-16T08:44:29.334 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42562.log 2024-04-16T08:44:29.335 INFO:teuthology.orchestra.run.smithi079.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45572.log.gz 2024-04-16T08:44:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35866.log 2024-04-16T08:44:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37984.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37984.log.gz 2024-04-16T08:44:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42562.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42562.log.gz 2024-04-16T08:44:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42226.log 2024-04-16T08:44:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43024.log 2024-04-16T08:44:29.335 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35866.log.gz 2024-04-16T08:44:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41302.log 2024-04-16T08:44:29.335 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42226.log.gz 2024-04-16T08:44:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38278.log 2024-04-16T08:44:29.336 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43024.log.gz 2024-04-16T08:44:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33039.log 2024-04-16T08:44:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41302.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41302.log.gz 2024-04-16T08:44:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38278.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38278.log.gz 2024-04-16T08:44:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42646.log 2024-04-16T08:44:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:gzip/var/log/ceph/ceph-client.admin.33039.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41386.log 2024-04-16T08:44:29.336 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33039.log.gz 2024-04-16T08:44:29.337 INFO:teuthology.orchestra.run.smithi079.stderr:gzip/var/log/ceph/ceph-client.admin.42646.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42184.log 2024-04-16T08:44:29.337 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42646.log.gz 2024-04-16T08:44:29.337 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31879.log 2024-04-16T08:44:29.337 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2024-04-16T08:44:29.337 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36919.log 2024-04-16T08:44:29.337 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42184.log.gz 2024-04-16T08:44:29.337 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31879.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31879.log.gz 2024-04-16T08:44:29.337 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34026.log 2024-04-16T08:44:29.337 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30718.log 2024-04-16T08:44:29.338 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36919.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36919.log.gz 2024-04-16T08:44:29.338 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34026.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34026.log.gz 2024-04-16T08:44:29.338 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44326.log 2024-04-16T08:44:29.338 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35824.log 2024-04-16T08:44:29.338 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30718.log.gz 2024-04-16T08:44:29.338 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44326.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44326.log.gz 2024-04-16T08:44:29.338 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44536.log 2024-04-16T08:44:29.338 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38950.log 2024-04-16T08:44:29.339 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35824.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35824.log.gz 2024-04-16T08:44:29.339 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44200.log 2024-04-16T08:44:29.339 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44536.log.gz 2024-04-16T08:44:29.339 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32278.log 2024-04-16T08:44:29.339 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38950.log.gz 2024-04-16T08:44:29.339 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45508.log 2024-04-16T08:44:29.339 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44200.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.44200.log.gz 2024-04-16T08:44:29.339 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29731.log 2024-04-16T08:44:29.339 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32278.log.gz 2024-04-16T08:44:29.340 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45508.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45508.log.gz 2024-04-16T08:44:29.340 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42016.log 2024-04-16T08:44:29.340 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31221.log 2024-04-16T08:44:29.340 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29731.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29731.log.gz 2024-04-16T08:44:29.340 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42016.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42016.log.gz 2024-04-16T08:44:29.340 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38530.log 2024-04-16T08:44:29.340 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40126.log 2024-04-16T08:44:29.340 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31221.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.31221.log.gz 2024-04-16T08:44:29.340 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34942.log 2024-04-16T08:44:29.341 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-04-16T08:44:29.341 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40126.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28673.log 2024-04-16T08:44:29.341 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40126.log.gz 2024-04-16T08:44:29.341 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32020.log 2024-04-16T08:44:29.341 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34942.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34942.log.gz 2024-04-16T08:44:29.341 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.tmp-client.admin.28673.log: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28673.log.gz 2024-04-16T08:44:29.341 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32634.log 2024-04-16T08:44:29.341 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34715.log 2024-04-16T08:44:29.341 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32020.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32020.log.gz 2024-04-16T08:44:29.342 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32634.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-16T08:44:29.342 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32634.log.gz 2024-04-16T08:44:29.342 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42142.log 2024-04-16T08:44:29.342 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34715.log.gz 2024-04-16T08:44:29.342 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31530.log 2024-04-16T08:44:29.342 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42142.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42142.log.gz 2024-04-16T08:44:29.342 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44242.log 2024-04-16T08:44:29.342 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32943.log 2024-04-16T08:44:29.342 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31530.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31530.log.gz 2024-04-16T08:44:29.343 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44242.log: gzip 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44242.log.gz 2024-04-16T08:44:29.343 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36541.log 2024-04-16T08:44:29.343 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5/var/log/ceph/ceph-client.admin.32943.log: --verbose -- /var/log/ceph/ceph-client.admin.38572.log 2024-04-16T08:44:29.343 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32943.log.gz 2024-04-16T08:44:29.343 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36541.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36541.log.gz 2024-04-16T08:44:29.343 INFO:teuthology.orchestra.run.smithi079.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38656.log 2024-04-16T08:44:29.343 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32803.log 2024-04-16T08:44:29.343 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38572.log.gz 2024-04-16T08:44:29.344 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37477.log 2024-04-16T08:44:29.344 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38656.log.gz 2024-04-16T08:44:29.344 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32763.log 2024-04-16T08:44:29.344 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32803.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32803.log.gz 2024-04-16T08:44:29.344 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43528.log 2024-04-16T08:44:29.344 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37477.log.gz 2024-04-16T08:44:29.344 INFO:teuthology.orchestra.run.smithi079.stderr:gzip/var/log/ceph/ceph-client.admin.32763.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35120.log 49.6% 2024-04-16T08:44:29.344 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32763.log.gz 2024-04-16T08:44:29.345 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43528.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43528.log.gz 2024-04-16T08:44:29.345 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36499.log 2024-04-16T08:44:29.345 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37900.log 2024-04-16T08:44:29.345 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35120.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35120.log.gz 2024-04-16T08:44:29.345 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42310.log 2024-04-16T08:44:29.345 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36499.log.gz 2024-04-16T08:44:29.345 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42268.log 2024-04-16T08:44:29.345 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37900.log.gz 2024-04-16T08:44:29.345 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39474.log 2024-04-16T08:44:29.346 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42310.log.gz 2024-04-16T08:44:29.346 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45664.log 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42268.log.gz 2024-04-16T08:44:29.346 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-16T08:44:29.346 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39474.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39474.log.gz 2024-04-16T08:44:29.346 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-16T08:44:29.346 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45664.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45664.log.gz 2024-04-16T08:44:29.346 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34984.log 2024-04-16T08:44:29.346 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34764.log 2024-04-16T08:44:29.347 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.34984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38110.log 2024-04-16T08:44:29.347 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34984.log.gz 2024-04-16T08:44:29.347 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34895.log 2024-04-16T08:44:29.347 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34764.log.gz 2024-04-16T08:44:29.347 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40966.log 2024-04-16T08:44:29.347 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38110.log: /var/log/ceph/ceph-client.admin.34895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32118.log 2024-04-16T08:44:29.347 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% 48.4% -- replaced with /var/log/ceph/ceph-client.admin.34895.log.gz 2024-04-16T08:44:29.347 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38110.log.gz 2024-04-16T08:44:29.347 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40966.log: 90.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-16T08:44:29.348 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.40966.log.gz 2024-04-16T08:44:29.348 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41890.log 2024-04-16T08:44:29.348 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36751.log 2024-04-16T08:44:29.348 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32118.log.gz 2024-04-16T08:44:29.348 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37816.log 2024-04-16T08:44:29.348 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41890.log.gz 2024-04-16T08:44:29.348 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45387.log 2024-04-16T08:44:29.348 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36751.log.gz 2024-04-16T08:44:29.348 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37816.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37816.log.gz 2024-04-16T08:44:29.349 INFO:teuthology.orchestra.run.smithi079.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44158.log 2024-04-16T08:44:29.349 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41974.log 2024-04-16T08:44:29.349 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45387.log.gz 2024-04-16T08:44:29.349 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44158.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44158.log.gz 2024-04-16T08:44:29.349 INFO:teuthology.orchestra.run.smithi079.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41596.log 2024-04-16T08:44:29.349 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41974.log: gzip 49.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.45911.log 2024-04-16T08:44:29.349 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41974.log.gz 2024-04-16T08:44:29.349 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41596.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41596.log.gz 2024-04-16T08:44:29.349 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36244.log 2024-04-16T08:44:29.350 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44452.log 2024-04-16T08:44:29.350 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45911.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.45911.log.gz 2024-04-16T08:44:29.350 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36244.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36244.log.gz 2024-04-16T08:44:29.350 INFO:teuthology.orchestra.run.smithi079.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42940.log 2024-04-16T08:44:29.350 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38404.log 2024-04-16T08:44:29.350 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44452.log.gz 2024-04-16T08:44:29.350 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42940.log: gzip -5 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42940.log.gz 2024-04-16T08:44:29.350 INFO:teuthology.orchestra.run.smithi079.stderr: --verbose -- /var/log/ceph/ceph-client.0.35014.log 2024-04-16T08:44:29.351 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45208.log 2024-04-16T08:44:29.351 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38404.log.gz 2024-04-16T08:44:29.351 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.0.35014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40336.log 2024-04-16T08:44:29.351 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45208.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45208.log.gz 2024-04-16T08:44:29.351 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41470.log 2024-04-16T08:44:29.351 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40336.log: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45124.log 2024-04-16T08:44:29.351 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40336.log.gz 2024-04-16T08:44:29.351 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36667.log 2024-04-16T08:44:29.351 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41470.log.gz 2024-04-16T08:44:29.352 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45124.log: -- /var/log/ceph/ceph-client.admin.42730.log 2024-04-16T08:44:29.352 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45124.log.gz 2024-04-16T08:44:29.352 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36667.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36667.log.gz 2024-04-16T08:44:29.352 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30552.log 2024-04-16T08:44:29.352 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42730.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42730.log.gz 2024-04-16T08:44:29.352 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39916.log 2024-04-16T08:44:29.352 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30552.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30552.log.gz 2024-04-16T08:44:29.352 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33777.log 2024-04-16T08:44:29.352 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39916.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32456.log 2024-04-16T08:44:29.353 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39916.log.gz 2024-04-16T08:44:29.353 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31142.log 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33777.log.gz 2024-04-16T08:44:29.353 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-16T08:44:29.353 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32456.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32456.log.gz 2024-04-16T08:44:29.353 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44704.log 2024-04-16T08:44:29.353 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31142.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31142.log.gz 2024-04-16T08:44:29.353 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29966.log 2024-04-16T08:44:29.353 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44704.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44704.log.gz 2024-04-16T08:44:29.354 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34493.log 2024-04-16T08:44:29.354 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.29966.log: /var/log/ceph/ceph-client.admin.40294.log 2024-04-16T08:44:29.354 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29966.log.gz 2024-04-16T08:44:29.354 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34493.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34493.log.gz 2024-04-16T08:44:29.354 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45756.log 2024-04-16T08:44:29.354 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40294.log: -- /var/log/ceph/ceph-client.admin.43738.log 2024-04-16T08:44:29.354 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40294.log.gz/var/log/ceph/ceph-client.admin.45756.log: 2024-04-16T08:44:29.354 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45756.log.gz 2024-04-16T08:44:29.354 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36076.log 2024-04-16T08:44:29.355 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45340.log 2024-04-16T08:44:29.355 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43738.log: /var/log/ceph/ceph-client.admin.36076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30660.log 2024-04-16T08:44:29.355 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43738.log.gz 2024-04-16T08:44:29.355 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36076.log.gz 2024-04-16T08:44:29.355 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45340.log: 8.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45541.log 2024-04-16T08:44:29.355 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45340.log.gz 2024-04-16T08:44:29.355 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33088.log 2024-04-16T08:44:29.355 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30660.log.gz 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41554.log 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45541.log: /var/log/ceph/ceph-client.admin.33088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44410.log 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.45541.log.gz 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33088.log.gz 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41554.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41554.log.gz 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32675.log 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44410.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44410.log.gz 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36457.log 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32675.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41176.log 2024-04-16T08:44:29.356 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32675.log.gz 2024-04-16T08:44:29.357 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41008.log 2024-04-16T08:44:29.357 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36457.log.gz 2024-04-16T08:44:29.357 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34395.log 2024-04-16T08:44:29.357 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41176.log.gz 2024-04-16T08:44:29.357 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39748.log 2024-04-16T08:44:29.357 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41008.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41008.log.gz 2024-04-16T08:44:29.357 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35160.log 2024-04-16T08:44:29.357 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39748.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39748.log.gz 2024-04-16T08:44:29.357 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44074.log 2024-04-16T08:44:29.358 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34395.log: 50.0%/var/log/ceph/ceph-client.admin.35160.log: -- replaced with /var/log/ceph/ceph-client.admin.34395.log.gz 2024-04-16T08:44:29.358 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35160.log.gz 2024-04-16T08:44:29.358 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44074.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44074.log.gz 2024-04-16T08:44:29.358 INFO:teuthology.orchestra.run.smithi079.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.0.35014.log.gz 2024-04-16T08:44:29.359 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3335 16 1935 1919 0.0359585 0 - 0.363217 2024-04-16T08:44:29.406 INFO:teuthology.orchestra.run.smithi079.stderr: 89.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-16T08:44:29.495 INFO:teuthology.orchestra.run.smithi079.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.0.31310.log.gz 2024-04-16T08:44:30.311 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3336 16 1935 1919 0.0359477 0 - 0.363217 2024-04-16T08:44:31.311 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3337 16 1935 1919 0.035937 0 - 0.363217 2024-04-16T08:44:31.702 INFO:teuthology.orchestra.run.smithi167.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-16T08:44:31.902 INFO:teuthology.orchestra.run.smithi079.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-16T08:44:31.998 INFO:teuthology.orchestra.run.smithi167.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-16T08:44:32.000 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-16T08:44:32.000 INFO:teuthology.orchestra.run.smithi167.stderr:real 0m2.866s 2024-04-16T08:44:32.000 INFO:teuthology.orchestra.run.smithi167.stderr:user 0m5.248s 2024-04-16T08:44:32.000 INFO:teuthology.orchestra.run.smithi167.stderr:sys 0m0.245s 2024-04-16T08:44:32.186 INFO:teuthology.orchestra.run.smithi079.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-16T08:44:32.188 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-16T08:44:32.188 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m3.060s 2024-04-16T08:44:32.188 INFO:teuthology.orchestra.run.smithi079.stderr:user 0m6.186s 2024-04-16T08:44:32.189 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m0.393s 2024-04-16T08:44:32.189 INFO:tasks.ceph:Archiving logs... 2024-04-16T08:44:32.189 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962/remote/smithi079/log 2024-04-16T08:44:32.190 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T08:44:32.336 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3338 16 1935 1919 0.0359262 0 - 0.363217 2024-04-16T08:44:33.326 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3339 16 1935 1919 0.0359154 0 - 0.363217 2024-04-16T08:44:33.532 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962/remote/smithi167/log 2024-04-16T08:44:33.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T08:44:34.325 INFO:tasks.radosbench.radosbench.0.smithi079.stdout:2024-04-16T08:44:34.313467+0000 min lat: 0.0163555 max lat: 31.0929 avg lat: 0.363217 2024-04-16T08:44:34.325 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-16T08:44:34.325 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3340 16 1935 1919 0.0359047 0 - 0.363217 2024-04-16T08:44:34.555 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T08:44:34.574 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T08:44:34.584 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_30b0aa0009a925d6a46536e55c5656edcd51ecdb/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-16T08:44:34.639 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T08:44:34.640 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T08:44:34.643 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T08:44:34.720 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T08:44:34.728 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T08:44:34.728 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T08:44:34.730 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T08:44:34.745 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-04-16T08:44:34.747 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-04-16T08:44:34.751 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T08:44:34.751 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-04-16T08:44:34.751 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +1396us[+1393us] +/- 57ms 2024-04-16T08:44:34.751 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 -1286us[-1286us] +/- 54ms 2024-04-16T08:44:34.751 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -457us[ -461us] +/- 82ms 2024-04-16T08:44:34.752 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T08:44:34.753 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T08:44:34.753 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-04-16T08:44:34.753 INFO:teuthology.orchestra.run.smithi167.stdout:^* hv01.front.sepia.ceph.com 3 7 377 24 +1465us[+1411us] +/- 57ms 2024-04-16T08:44:34.754 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -1280us[-1333us] +/- 54ms 2024-04-16T08:44:34.754 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 87 -439us[ -484us] +/- 81ms 2024-04-16T08:44:34.754 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T08:44:34.754 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T08:44:34.768 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T08:44:34.768 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T08:44:34.788 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T08:44:34.871 DEBUG:teuthology.orchestra.run.smithi167:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T08:44:34.943 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T08:44:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T08:44:34.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T08:44:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T08:44:35.009 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T08:44:35.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T08:44:35.055 INFO:teuthology.task.internal:Duration was 4312.921181 seconds 2024-04-16T08:44:35.056 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T08:44:35.088 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T08:44:35.089 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T08:44:35.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T08:44:35.135 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T08:44:35.140 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T08:44:35.311 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3341 16 1935 1919 0.0358939 0 - 0.363217 2024-04-16T08:44:35.377 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T08:44:35.377 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-04-16T08:44:35.377 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T08:44:35.397 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2024-04-16T08:44:35.397 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T08:44:35.453 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T08:44:35.454 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T08:44:35.456 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T08:44:35.487 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T08:44:35.487 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T08:44:35.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T08:44:35.644 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T08:44:35.654 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T08:44:35.654 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T08:44:35.683 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T08:44:35.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T08:44:35.721 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T08:44:35.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T08:44:35.749 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-04-16T08:44:35.779 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2024-04-16T08:44:35.803 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T08:44:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T08:44:35.832 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T08:44:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T08:44:35.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T08:44:35.867 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T08:44:35.868 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962/remote/smithi079 2024-04-16T08:44:35.868 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T08:44:35.910 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962/remote/smithi167 2024-04-16T08:44:35.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T08:44:35.946 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T08:44:35.946 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T08:44:35.948 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T08:44:35.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T08:44:36.008 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T08:44:36.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T08:44:36.031 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T08:44:36.031 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T08:44:36.033 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T08:44:36.046 INFO:teuthology.orchestra.run.smithi079.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 08:44 /home/ubuntu/cephtest 2024-04-16T08:44:36.052 INFO:teuthology.orchestra.run.smithi167.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 08:44 /home/ubuntu/cephtest 2024-04-16T08:44:36.053 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T08:44:36.063 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T08:44:36.099 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-16T08:44:36.099 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T08:44:36.123 DEBUG:teuthology.nuke:shortname: smithi079 2024-04-16T08:44:36.124 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T08:44:36.144 DEBUG:teuthology.nuke:shortname: smithi167 2024-04-16T08:44:36.144 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T08:44:36.147 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:08.204180', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA5UD4iWmcHqD0asquxePKd1zDgMEmVjKTp6tq2G1cvMf5S0DQQU9rBSKz2/86Sc2+mdSYsp1sjTz8YVmxtua0Y='} 2024-04-16T08:44:36.168 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:22:04-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657962', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:23:08.202417', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvF4CbbQpotyRwsdFpVtdhZM8X38sysgVEwOX2Waa1gG2kMjIKBCYqQIJe2ufBRAsWXlIguOU6QCW3lf1wy3uI='} 2024-04-16T08:44:36.186 INFO:teuthology.orchestra.console.smithi079:Power off 2024-04-16T08:44:36.186 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T08:44:36.211 DEBUG:teuthology.orchestra.console.smithi079:power off output: Chassis Power Control: Down/Off 2024-04-16T08:44:36.211 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T08:44:36.226 INFO:teuthology.orchestra.console.smithi167:Power off 2024-04-16T08:44:36.226 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T08:44:36.236 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-04-16T08:44:36.248 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2024-04-16T08:44:36.249 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T08:44:36.273 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-04-16T08:44:36.312 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3342 16 1935 1919 0.0358832 0 - 0.363217 2024-04-16T08:44:37.312 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3343 16 1935 1919 0.0358725 0 - 0.363217 2024-04-16T08:44:38.314 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3344 16 1935 1919 0.0358617 0 - 0.363217 2024-04-16T08:44:39.312 INFO:tasks.radosbench.radosbench.0.smithi079.stdout: 3345 16 1935 1919 0.035851 0 - 0.363217 2024-04-16T08:44:40.237 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T08:44:40.274 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T08:44:40.348 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-04-16T08:44:40.384 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-04-16T08:44:44.353 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T08:44:44.385 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T08:44:44.465 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is off 2024-04-16T08:44:44.495 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-04-16T08:44:44.566 INFO:teuthology.orchestra.console.smithi079:Power off completed 2024-04-16T08:44:48.496 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T08:44:48.608 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-04-16T08:44:48.709 INFO:teuthology.orchestra.console.smithi167:Power off completed 2024-04-16T08:44:48.813 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} duration: 4312.921181440353 failure_reason: reached maximum tries (501) after waiting for 3000 seconds flavor: crimson owner: scheduled_matan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=084a54f686b64bfe9bc6f7f885aa0dc2 status: fail success: false 2024-04-16T08:44:48.814 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T08:44:48.901 INFO:teuthology.run:FAIL